[gdal-dev] Time in GDAL

[Even Rouault]

Hi Edzer,

> Currently, parsing this from the metadata strings is
> possible but messy and ad hoc, error prone, and has to be done by each
> client for every driver.

Agreed that metadata handling can be messy and is a bit in freestyle mode 
between drivers. In GDAL 2.0, Dmitry introduced metadata readers and a 
formalism to standardize a few common ones. This is mentionned in the "IMAGERY 
Domain (remote sensing)" paragraph of http://gdal.org/gdal_datamodel.html . As 
this is short, I put it inline 
"""
For satellite or aerial imagery the IMAGERY Domain may be present. It depends 
on the existence of special metadata files near the image file. The files at 
the same directory with image file tested by the set of metadata readers, if 
files can be processed by the metadata reader, it fill the IMAGERY Domain with 
the following items:

    SATELLITEID: A satellite or scanner name
    CLOUDCOVER: Cloud coverage. The value between 0 - 100 or 999 if not 
available
    ACQUISITIONDATETIME: The image acquisition date time in UTC
"""

A few examples from the autotest suite

$ gdalinfo ../autotest/gcore/data/md_dg.tif -mdd imagery
[...]
Metadata (imagery):
  ACQUISITIONDATETIME=2010-04-01 12:00:00
  CLOUDCOVER=2
  SATELLITEID=WV03

$ gdalinfo ../autotest/gcore/data/md_re.tif -mdd imagery
[...]
Metadata (imagery):
  ACQUISITIONDATETIME=2010-02-01 12:00:00
  CLOUDCOVER=0
  SATELLITEID=RE-0

$ gdalinfo ../autotest/gcore/data/md_ls_b1.tif -mdd imagery
[...]
  ACQUISITIONDATETIME=2015-09-01 17:00:00
  CLOUDCOVER=1
  SATELLITEID=LANDSAT_8


This has been implement currently for the GTiff driver with side car files 
that contain metadata and whose format depend on each imagery provider (as 
well as in TIL and DIMAP drivers). There are currently readers for ALOS, 
DigitalGlobe, Eros, GeoEye, Kompsat, Landsat, OrbView, Pleiades, RapidEye, 
Resurs-DK1 and Spot.

So I can imagine we could start from this, extend it to other drivers where 
that make sense, and have distinct items when start time != end time, or just 
consider that ACQUISITIONDATETIME is the start time, and add a 
ENDACQUISITIONDATETIME.

You mention above "GetStartTime() and GetEndTime(), which operate (at least) 
on band". Do you think per band is needed ? The above approach is per dataset, 
but the GetMetadata() API is also available on bands, so that could be 
replicated there too if per-band timestamps are needed.

> ## Multiple time stamps
> 
> Datasets might have many time stamps, e.g. some related to the
> processing steps that a dataset has undergone. Other datasets, e.g.
> forecast data, may have two relevant times (two-dimensional time): the
> time at which the forecast was made (t0), and the forecast times the
> band refers to (e.g. t0+6h, t0+12h, t0+18h, t0+24h etc). For both cases,
> I believe there is a "default time": the time of observation or
> prediction a band refers to. Access to other time aspects could be
> obtained by tags as in GetStartTime(..., reference =
> "TIME_OF_REPROCESSING"), which might be driver dependent.

Yes, for the GRIB2 driver, we have (actually per-band) metadata like:
    GRIB_FORECAST_SECONDS=68400 sec
    GRIB_REF_TIME=  1203613200 sec UTC
    GRIB_VALID_TIME=  1203681600 sec UTC

GRIB_REF_TIME is your t0 / the time at which observations have been collected, 
and GRIB_VALID_TIME the time of the forecast.
The GRIB_REF_TIME could be potentially normalized as the ACQUISITIONDATETIME 
and the GRIB_VALID_TIME as a FORECASTDATETIME


> 
> ## NetCDF/udunits time
> 
> Most file formats will have time strings like 2017-03-14T10:40:11.026Z
> that should be pretty straightforward to handle. NetCDF however uses
> time encoded in a form understood by udunits2 [4]; for example band
> metadata may have
> 
>  time#units=days since 1978-01-01 00:00:00
>  NETCDF_DIM_time=1339
> 
> which refers to 1339 days after 1978-01-01 00:00:00. Since the units can
> be set very flexible, for such data I think one should either:
> (i) have GDAL link (optionally) to udunits2, and if present, use the
> library to convert to OFTDateTime;
> (ii) do not try this but return the time units as string and the time as
> double, and leave the conversion to the client.

(iii) if there are not so many variants of expressing the time (I can't 
imagine there are hundreds, and as GDAL has already "a couple" dependencies, 
better avoid adding new ones if we can avoid), we could also hard code the 
normalization in the netCDF driver.

> 
> 
> I would like to know:
> - whether there is support for this idea, in general (time in gdal),

I do think this makes sense

> - whether the approach sketched above makes sense

I'm not completely sure if we really need a dedicated method as you suggested, 
or if we can do with the normalized metadata approach that was introduced with 
the IMAGERY metadata domain.

> - what I've overlooked, what are the potential road blocks

Something not covered is the writing side, ie preservation of those metadata 
when typically gdal_translat'ing. But this can be considered out of scope 
currently. Potentially drivers could try to parse the IMAGERY metadata domain 
of the source dataset and do their best to encode the items into the natural 
way of the output driver.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com