[gdal-dev] Can't read HDF4 - TRMM
Jason Roberts
jason.roberts at duke.edu
Tue May 25 11:54:23 EDT 2010
Ivan,
This sounds very much like the classic problem with HDF and netCDF where the
formats are extremely flexible and allow data to be stored in a variety of
ways, but the data providers could not all agree on the same way of doing
it. Different providers adopted different conventions such as HDF-EOS. GDAL
is designed to work with some products but not all of them (see
http://www.gdal.org/frmt_hdf4.html, look under "Georeference").
The solution to this might be to modify the HDF driver to recognize and read
the particular HDF "attributes" that are used in this particular file. For
example, it could look for dimensions called "longitude" and "latitude" and
then assume they are x and y respectively. But this approach can get very
tedious: do you look for "longitude", "Longitude", "lon", "LON", "long",
etc? I have seen at least all of those. Where do you stop?
Perhaps the maintainers of the GDAL HDF driver could comment on their
general philosophy of recognizing different attributes or conventions.
Because this can be very tedious, they might not want to implement a
solution for every HDF that someone happens to find, but perhaps if you
prepared a patch that works for your particular HDFs, they would accept it.
I have no idea and am curious about their strategy.
pyhdf can work with any HDF but that is because it delegates the job of
recognizing how to work with the data to you, the programmer. You have to
recognize that "longitude" is the x dimension, for example. Thus it is not
really fair to compare pyhdf with GDAL because GDAL attempts to do that job
itself.
Hope this helps... (Sorry if this is all obvious...)
Best,
Jason
-----Original Message-----
From: gdal-dev-bounces at lists.osgeo.org
[mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Ivan Lucena
Sent: Tuesday, May 25, 2010 11:41 AM
To: Ivan Lucena; Joaquim Luis
Cc: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Can't read HDF4 - TRMM
It look like pyHDF know more about the HDF SD dimensions than the other
software, including GDAL:
>>> from pyhdf.SD import *
>>> d = SD('D:\\tmp\\3B430501016.HDF')
>>> d.datasets()
{'precipitation': (('scan', 'longitude', 'latitude'), (1, 1440, 400), 5, 0),
'relativeError': (('scan', 'longitude', 'latitude'), (1, 1440, 400), 5, 1)}
>>> ds = d.select('precipitation')
>>> ds.dimensions()
{'latitude': 400, 'longitude': 1440, 'scan': 1}
>>>
That means 1440 columns, 400 rows, and 1 band.
But GDAL, Idrisi, ArcGIS, Erdas and Envi reads it as 400 columns, 1440 rows,
and 1 band.
D:\BugsJar>gdal_translate HDF4_SDS:UNKNOWN:"3B430501016.HDF":0 out.tif
Input file size is 400, 1440
0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdalinfo out.tif
Driver: GTiff/GeoTIFF
Files: out.tif
Size is 400, 1440
Coordinate System is `'
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 1440.0)
Upper Right ( 400.0, 0.0)
Lower Right ( 400.0, 1440.0)
Center ( 200.0, 720.0)
Band 1 Block=400x5 Type=Float32, ColorInterp=Gray
There is something wrong with the internal documentation of that file but it
seems to be possible to find the dimensions from other HDF (not EOS) API
functions. I would try to include calls to SDdiminfo() or SDgetdimstr()
perhaps.
Regards,
Ivan
> -------Original Message-------
> From: Ivan Lucena <ivan.lucena at pmldnet.com>
> To: Joaquim Luis <jluis at ualg.pt>
> Cc: gdal-dev at lists.osgeo.org <gdal-dev at lists.osgeo.org>
> Subject: Re: [gdal-dev] Can't read HDF4 - TRMM
> Sent: May 24 '10 13:51
>
> Hi Joaquim,
>
> It seems to me that the HDF4 driver is taking the liberty of using the
EOS API to read that file that is actually in HDF format, more specifically
a SD (Scientific Dataset).
>
> http://trac.osgeo.org/gdal/browser/trunk/gdal/frmts/hdf4/hdf4dataset.cpp#
L762
>
> That is why it doesn't go to that call to SDfileinfo():
>
> http://trac.osgeo.org/gdal/browser/trunk/gdal/frmts/hdf4/hdf4dataset.cpp#
L991
>
> If change poDS->bIsHDFEOS to 0 in debugging mode and the subdatasets
showed up:
>
> Driver: HDF4/Hierarchical Data Format Release 4
> Files: 3B430501016.HDF
> Size is 512, 512
> Coordinate System is `'
> Subdatasets:
> SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"3B430501016.HDF":0
> SUBDATASET_1_DESC=[1x1440x400] precipitation (32-bit floating-point)
> SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"3B430501016.HDF":1
> SUBDATASET_2_DESC=[1x1440x400] relativeError (32-bit floating-point)
> Corner Coordinates:
> Upper Left ( 0.0, 0.0)
> Lower Left ( 0.0, 512.0)
> Upper Right ( 512.0, 0.0)
> Lower Right ( 512.0, 512.0)
> Center ( 256.0, 256.0)
>
> Now I can gdal_translate it to GTIFF but the image comes rotated by 90
degrees just like in other software.
>
> I think we can file a ticket to deal with the miss identification, at
least.
>
> Obrigado pela dica.
>
> Thanks for the tip,
>
> Ivan
>
> > -------Original Message-------
> > From: Joaquim Luis <jluis at ualg.pt>
> > To: Ivan Lucena <ivan.lucena at pmldnet.com>
> > Cc: gdal-dev at lists.osgeo.org <gdal-dev at lists.osgeo.org>
> > Subject: Re: [gdal-dev] Can't read HDF4 - TRMM
> > Sent: May 24 '10 10:46
> >
> > Ivan,
> >
> > Since nasa is now imitating me (http://mirador....) I gave it a shot
> > with Mirone, which obviously also failed because because it relies
> > mostly in gdal to read the hdf files.
> > My experience with the nasa satellite files is that they are a big
mess
> > with the metadata written in endless different ways which makes the
> > parsing very painful. There is some reference info in the files. See
> > what I got using Matlab (look at the end of this long list)
> >
> > Joaquim
> >
> >
> > >> info.Attributes(1).Value
> >
> > ans =
> >
> > OBJECT=OrbitNumber;
> > Value=-9999;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=OrbitNumber;
> >
> > OBJECT=RangeBeginningDate;
> > Value=2010/04/01;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=RangeBeginningDate;
> >
> > OBJECT=RangeBeginningTime;
> > Value=00:00:00;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=RangeBeginningTime;
> >
> > OBJECT=RangeEndingDate;
> > Value=2010/05/01;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=RangeEndingDate;
> >
> > OBJECT=RangeEndingTime;
> > Value=00:00:00;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=RangeEndingTime;
> >
> > OBJECT=GranulePointer;
> > Value="3B43.100401.6A.HDF";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=GranulePointer;
> >
> > OBJECT=ShortName;
> > Value="Surface Rain from all Satellite and Surface";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=ShortName;
> >
> > OBJECT=SizeMBECSDataGranule;
> > Value=4.394;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=SizeMBECSDataGranule;
> >
> > OBJECT=LongitudeOfMaximumLatitude;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=LongitudeOfMaximumLatitude;
> >
> > OBJECT=SpatialCoverageType;
> > Value="Horizontal";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=SpatialCoverageType;
> >
> > OBJECT=EllipsoidName;
> > Value="NULL";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=EllipsoidName;
> >
> > OBJECT=EquatorialRadius;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=EquatorialRadius;
> >
> > OBJECT=DenominatorFlatteningRatio;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=DenominatorFlatteningRatio;
> >
> > OBJECT=OrbitalModelName;
> > Value="NULL";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=OrbitalModelName;
> >
> > OBJECT=SemiMajorAxis;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=SemiMajorAxis;
> >
> > OBJECT=MeanAnomaly;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=MeanAnomaly;
> >
> > OBJECT=RightAscensionNode;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=RightAscensionNode;
> >
> > OBJECT=ArgumentOfPerigee;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=ArgumentOfPerigee;
> >
> > OBJECT=Eccentricity;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=Eccentricity;
> >
> > OBJECT=Inclination;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=Inclination;
> >
> > OBJECT=EpochTime;
> > Value=99:99:99;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=EpochTime;
> >
> > OBJECT=EpochDate;
> > Value=9999/99/99;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=EpochDate;
> >
> > OBJECT=EpochMillisec;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=EpochMillisec;
> >
> > OBJECT=WestBoundingCoordinate;
> > Value=-180;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=WestBoundingCoordinate;
> >
> > OBJECT=EastBoundingCoordinate;
> > Value=180;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=EastBoundingCoordinate;
> >
> > OBJECT=NorthBoundingCoordinate;
> > Value=50;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=NorthBoundingCoordinate;
> >
> > OBJECT=SouthBoundingCoordinate;
> > Value=-50;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=SouthBoundingCoordinate;
> >
> > OBJECT=CenterLatitude;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=CenterLatitude;
> >
> > OBJECT=CenterLongitude;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=CenterLongitude;
> >
> > OBJECT=RadiusValue;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=RadiusValue;
> >
> > OBJECT=LatitudeResolution;
> > Value=".25deg";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=LatitudeResolution;
> >
> > OBJECT=LongitudeResolution;
> > Value=".25deg";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=LongitudeResolution;
> >
> > OBJECT=GeographicCoordinateUnits;
> > Value="Decimal Degrees";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=GeographicCoordinateUnits;
> >
> > OBJECT=TemporalRangeType;
> > Value="Continuous Range";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=TemporalRangeType;
> >
> > OBJECT=QualityAssuranceParameterName;
> > Value="ScienceQualityFlag";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=QualityAssuranceParameterName;
> >
> > OBJECT=QualityAssuranceParameterValue;
> > Value="NOT BEING INVESTIGATED";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=QualityAssuranceParameterValue;
> >
> > OBJECT=ReprocessingActual;
> > Value="NULL";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=ReprocessingActual;
> >
> > OBJECT=BrowsePointer;
> > Value="3B43_BR.100401.6A.PNG";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=BrowsePointer;
> >
> > OBJECT=ScienceContact;
> > Value="George Huffman";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=ScienceContact;
> >
> > OBJECT=MeanMotion;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=MeanMotion;
> >
> > OBJECT=OrbitAdjustFlag;
> > Value=-9999;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=OrbitAdjustFlag;
> >
> > OBJECT=AttitudeModeFlag;
> > Value=-9999;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=AttitudeModeFlag;
> >
> > OBJECT=SolarBetaAngleAtBeginningOfGranule;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=SolarBetaAngleAtBeginningOfGranule;
> >
> > OBJECT=SolarBetaAngleAtEndOfGranule;
> > Value=-9999.9;
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=SolarBetaAngleAtEndOfGranule;
> >
> > OBJECT=SensorAlignment;
> > Value="NULL";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=SensorAlignment;
> >
> > OBJECT=SensorAlignmentChannelOffsets;
> > Value="NULL";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=SensorAlignmentChannelOffsets;
> >
> > OBJECT=ScanPathModel;
> > Value="NULL";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=ScanPathModel;
> >
> > OBJECT=ScanPathModelParam;
> > Value="NULL";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=ScanPathModelParam;
> >
> > OBJECT=EphemerisFileID;
> > Value="NULL";
> > Data_Location=PGE;
> > Mandatory=FALSE;
> > END_OBJECT=EphemerisFileID;
> >
> >
> >
> >
> > > Hi,
> > >
> > > I ran gdalinfo on some HDF4 files and no subdataset shows up.
> > >
> > > The data is available for download at this website:
http://mirador.gsfc.nasa.gov/cgi-bin/mirador/presentNavigation.pl?tree=proje
ct&project=TRMM&dataGroup=Gridded&dataset=3B43:%20Monthly%200.25%20x%200.25%
20degree%20merged%20TRMM%20and%20other%20sources%20estimates&version=006
> > >
> > > I tried reading those some file on:
> > >
> > > - Idrisi
> > > - ArcGIS 9.3
> > > - ArcGIS 10 (RC)
> > > - Erdas
> > >
> > > They do identify the internal datasets but they cannot identify the
reference system and the images they are flipped by 90' clockwise. But that
is not due to rotation. I would guess. It is more like a misinterpretation
of the dimension parameter, taking rows as columns and columns as rows. It
looks like the documentation for that particular product is missing here:
http://mirador.gsfc.nasa.gov/cgi-bin/mirador/servcoll.pl?helpmenuclass=inven
tory&SearchButton=Search%20GES-DISC
> > >
> > > Does anybody has a clue?
> > >
> > > BTW. There is a netCDF version of that data that also has some
issues.
> > >
> > > Regards,
> > >
> > > Ivan
> > >
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > gdal-dev mailing list
> > > gdal-dev at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > >
> > >
> > >
> >
> >
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
More information about the gdal-dev
mailing list