<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Why not read all date/time data from records as accurate as
possible? For example for OFTDate we get date by <a class="el"
href="http://www.gdal.org/classOGRFeature.html#a6c5d2444407b07e07b79863c42ee7a49">GetFieldAsDateTime</a>
and time is zero.<br>
It's strange to analyse data structure during reading the records as
we already have field definition. <br>
We can use for old datasets type DateTime + SubType ODTP_YMDHMSm and
new datasets let the user to choose the subtype. Certainly some
formats support this new type + subtype now (i.e. Postgres/PostGIS,
etc.).<br>
<br>
Postgres data type mapping:<br>
<tt class="TYPE">date -> OFTDateTime + </tt>ODTP_YMD<br>
<tt class="TYPE">time</tt><tt class="TYPE"> -> OFTDateTime + </tt>ODTP_HMS<br>
<tt class="TYPE">timestamp -> </tt><tt class="TYPE">OFTDateTime
+ </tt>ODTP_YMDHMSm
<pre class="moz-signature" cols="72">Best regards,
Dmitry</pre>
<div class="moz-cite-prefix">07.04.2015 00:14, Even Rouault пишет:<br>
</div>
<blockquote cite="mid:201504062314.57392.even.rouault@spatialys.com"
type="cite">
<pre wrap="">Le lundi 06 avril 2015 23:11:21, Dmitriy Baryshnikov a écrit :
</pre>
<blockquote type="cite">
<pre wrap="">Hi Even,
It seems to me that this is duplicating of RFC 50: OGR field subtypes.
For example we have the master field type DateTime and Subtype - Year.
So the internal structure for date/time representation may be adopt to
such technique.
</pre>
</blockquote>
<pre wrap="">
The subtype is defined at field definition level. In all formats we currently
handle we only know the date/time precision when reading values (and they
might have different precision between records), so after having created the
layer and field definitions.
</pre>
<blockquote type="cite">
<pre wrap="">
Best regards,
Dmitry
06.04.2015 15:02, Even Rouault пишет:
</pre>
<blockquote type="cite">
<pre wrap="">Le lundi 06 avril 2015 13:48:47, Even Rouault a écrit :
</pre>
<blockquote type="cite">
<pre wrap="">Le lundi 06 avril 2015 11:32:33, Dmitriy Baryshnikov a écrit :
</pre>
<blockquote type="cite">
<pre wrap="">The first solution looks reasonable. But there is lack in precision
field - there the only time is significant:
ODTP_HMSm
ODTP_HMS
ODTP_HM
ODTP_H
</pre>
</blockquote>
<pre wrap="">
As I didn't want to multiply the values in the enumeration, my intent
was to reuse the ODTP_YMDxxxx values for OFTTime only.
</pre>
</blockquote>
<pre wrap="">
I meant "for OFTTime too"
</pre>
<blockquote type="cite">
<pre wrap="">This was what I wanted
to intend with the precision between parenthesis in the comment of
ODTP_YMDH "Year, month, day (if OFTDateTime) and hour"
Or perhaps, the enumeration should capture the most precise part of the
(date)time structure ?
ODTP_Year
ODTP_Month
ODTP_Day
ODTP_Hour
ODTP_Minute
ODTP_Second
ODTP_Millisecond
</pre>
<blockquote type="cite">
<pre wrap="">etc.
Best regards,
Dmitry
05.04.2015 22:25, Even Rouault пишет:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
In an effort of revisiting <a class="moz-txt-link-freetext" href="http://trac.osgeo.org/gdal/ticket/2680">http://trac.osgeo.org/gdal/ticket/2680</a>,
which is about lack of precision of the current datetime structure,
I've imagined different solutions how to modify the OGRField
structure, and failed to pick up one that would be the obvious
solution, so opinions are welcome.
The issue is how to add (at least) microsecond accuracy to the
datetime structure, as a few formats support it explicitely or
implicitely : MapInfo, GPX, Atom (GeoRSS driver), GeoPackage, SQLite,
PostgreSQL, CSV, GeoJSON, ODS, XLSX, KML (potentially GML too)...
Below a few potential solutions :
---------------------------------------
Solution 1) : Millisecond accuracy, second becomes a float
This is the solution I've prototyped.
typedef union {
[...]
struct {
GInt16 Year;
GByte Month;
GByte Day;
GByte Hour;
GByte Minute;
GByte TZFlag;
GByte Precision; /* value in OGRDateTimePrecision */
float Second; /* from 00.000 to 60.999 (millisecond
accuracy) */
} Date;
} OGRField
So sub-second precision is representing with a single precision
floating point number, storing both integral and decimal parts. (we
could theorically have a hundredth of millisecond accuracy, 10^-5 s,
since 6099999 fits on the 23 bits of the mantissa)
Another addition is the Precision field that indicates which parts of
the datetime structure are significant.
/** Enumeration that defines the precision of a DateTime.
* @since GDAL 2.0
*/
typedef enum
{
ODTP_Undefined, /**< Undefined */
ODTP_Guess, /**< Only valid when setting through
SetField(i,year,
month...) where OGR will guess */
ODTP_Y, /**< Year is significant */
ODTP_YM, /**< Year and month are significant*/
ODTP_YMD, /**< Year, month and day are significant */
ODTP_YMDH, /**< Year, month, day (if OFTDateTime) and
hour are
significant */
ODTP_YMDHM, /**< Year, month, day (if OFTDateTime), hour
and
minute are significant */
ODTP_YMDHMS, /**< Year, month, day (if OFTDateTime),
hour, minute
and integral second are significant */
ODTP_YMDHMSm, /**< Year, month, day (if OFTDateTime),
hour, minute
and second with microseconds are significant */
} OGRDateTimePrecision;
I think this is important since "2015/04/05 17:12:34" and "2015/04/05
17:12:34.000" do not really mean the same thing and it might be good
to be able to preserve the original accuracy when converting between
formats.
A drawback of this solution is that the size of the OGRField structure
increases from 8 bytes to 12 on 32 bit builds (and remain 16 bytes on
64 bit). This is probably not that important since in most cases not
that many OGRField structures are instanciated at one time (typically,
you iterate over features one at a time).
This could be more of a problem for use cases that involve the MEM
driver, as it keep all features in memory.
Another drawback is that the change of the structure might not be
directly noticed by application developers as the Second field name is
preserved, but a new Precision field is added, so there's a risk that
Precision is let uninitialized if the field is set through
OGRFeature::SetField(int iFieldIndex, OGRField* psRawField). That
could lead to unexpected formatting (but hopefully not crashes with
defensive programming). However I'd think it is unlikely that many
applications directly manipulate OGRField directly, instead of using
the getters and setters of OGRFeature.
---------------------------------------
Solution 2) : Millisecond accuracy, second and milliseconds in
distinct fields
typedef union {
[...]
struct {
GInt16 Year;
GByte Month;
GByte Day;
GByte Hour;
GByte Minute;
GByte TZFlag;
GByte Precision; /* value in OGRDateTimePrecision */
GByte Second; /* from 0 to 60 */
GUInt16 Millisecond; /* from 0 to 999 */
} Date;
} OGRField
Same size of structure as in 1)
---------------------------------------
Solution 3) : Millisecond accuracy, pack all fields
Conceptually, this would use bit fields to avoid wasting unused bits :
typedef union {
[...]
struct {
GInt16 Year;
GUIntBig Month:4;
GUIntBig Day:5;
GUIntBig Hour:5;
GUIntBig Minute:6;
GUIntBig Second:6;
GUIntBig Millisecond:10; /* 0-999 */
GUIntBig TZFlag:8;
GUIntBig Precision:4;
} Date;
} OGRField;
This was proposed in the above mentionned ticket. And as there were
enough remaining bits, I've also added the Precision field (and in all
other solutions).
The advantage is that sizeof(mydate) remains 8 bytes on 32 bits
builds.
But the C standard only defines bitfields of int/unsigned int, so this
is not portable, plus the fact that the way bits are packed is not
defined by the standard, so different compilers could come up with
different packing. A workaround is to do the bit manipulation through
macros :
typedef union {
[...]
struct {
GUIntBig opaque;
} Date;
} OGRField;
#define GET_BITS(x,y_bits,shift) (int)(((x).Date.opaque >>
(shift)) & ((1 << (y_bits))-1))
#define GET_YEAR(x) (short)GET_BITS(x,16,64-16)
#define GET_MONTH(x) GET_BITS(x,4,64-16-4)
#define GET_DAY(x) GET_BITS(x,5,64-16-4-5)
#define GET_HOUR(x) GET_BITS(x,5,64-16-4-5-5)
#define GET_MINUTE(x) GET_BITS(x,6,64-16-4-5-5-6)
#define GET_SECOND(x) GET_BITS(x,6,64-16-4-5-5-6-6)
#define GET_MILLISECOND(x) GET_BITS(x,10,64-16-4-5-5-6-6-10)
#define GET_TZFLAG(x) GET_BITS(x,8,64-16-4-5-5-6-6-10-8)
#define GET_PRECISION(x) GET_BITS(x,4,64-16-4-5-5-6-6-10-8-4)
#define SET_BITS(x,y,y_bits,shift) (x).Date.opaque = ((x).Date.opaque
& (~( (GUIntBig)((1 << (y_bits))-1) << (shift) )) | ((GUIntBig)(y) <<
(shift)))
#define SET_YEAR(x,val) SET_BITS(x,val,16,64-16)
#define SET_MONTH(x,val) SET_BITS(x,val,4,64-16-4)
#define SET_DAY(x,val) SET_BITS(x,val,5,64-16-4-5)
#define SET_HOUR(x,val) SET_BITS(x,val,5,64-16-4-5-5)
#define SET_MINUTE(x,val) SET_BITS(x,val,6,64-16-4-5-5-6)
#define SET_SECOND(x,val) SET_BITS(x,val,6,64-16-4-5-5-6-6)
#define SET_MILLISECOND(x,val)
SET_BITS(x,val,10,64-16-4-5-5-6-6-10) #define SET_TZFLAG(x,val)
SET_BITS(x,val,8,64-16-4-5-5-6-6-10-8) #define SET_PRECISION(x,val)
SET_BITS(x,val,4,64-16-4-5-5-6-6-10-8-4)
Main advantage: the size of OGRField remains unchanged (so 8 bytes on
32-bit builds).
Drawback: manipulation of datetime members is less natural, but there
are not that many places in the GDAL code base were the OGRField.Date
members are used, so it is not much that a problem.
---------------------------------------
Solution 4) : Microsecond accuracy with one field
Solution 1) used a float for second and sub-second, but a float has
only 23 bits of mantissa, which is enough to represent second with
millisecond accuracy, but not for microsecond (you need 26 bits for
that). So use a 32-bit integer instead of a 32-bit floating point.
typedef union {
[...]
struct {
GInt16 Year;
GByte Month;
GByte Day;
GByte Hour;
GByte Minute;
GByte TZFlag;
GByte Precision; /* value in OGRDateTimePrecision */
GUInt32 Microseconds; /* 00000000 to 59999999 */
} Date;
} OGRField
Same as solution 1: sizeof(OGRField) becomes 12 bytes on 32-bit builds
(and remain 16 bytes on 64-bit builds)
We would need to add an extra value in OGRDateTimePrecision to mean
the microsecond accuracy.
Not really clear we need microseconds accuracy... Most formats that
support subsecond accuracy use ISO 8601 representation (e.g. YYYY-MM-
DDTHH:MM:SS.sssssZ) that doesn't define the maximal number of decimals
beyond second. From
<a class="moz-txt-link-freetext" href="http://www.postgresql.org/docs/9.1/static/datatype-datetime.html">http://www.postgresql.org/docs/9.1/static/datatype-datetime.html</a>,
PostgreSQL supports microsecond accuracy.
---------------------------------------
Solution 5) : Microsecond with 3 fields
A variant where we split second into 3 integer parts:
typedef union {
[...]
struct {
GInt16 Year;
GByte Month;
GByte Day;
GByte Hour;
GByte Minute;
GByte TZFlag;
GByte Precision; /* value in OGRDateTimePrecision */
GByte Second; /* 0 to 59 */
GUInt16 Millisecond; /* 0 to 999 */
GUInt16 Microsecond; /* 0 to 999 */
} Date;
} OGRField
Drawback: due to alignment, sizeof(OGRField) becomes 16 bytes on
32-bit builds (and remain 16 bytes on 64-bit builds)
---------------------------------------
Solution 6) : Nanosecond accuracy and beyond !
Now that we are using 16 bytes, why not having nanosecond accuracy ?
typedef union {
[...]
struct {
GInt16 Year;
GByte Month;
GByte Day;
GByte Hour;
GByte Minute;
GByte TZFlag;
GByte Precision; /* value in OGRDateTimePrecision */
double Second; /* 0.000000000 to 60.999999999 */
} Date;
} OGRField
Actually we even have picosecond accuracy! (since for picoseconds, we
need 46 bits and a double has 52 bits of mantissa). And if we use a
64-bit integer instead of a double, we can have femtosecond accuracy
;-)
Any preference ?
Even
</pre>
</blockquote>
<pre wrap="">
_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="http://lists.osgeo.org/mailman/listinfo/gdal-dev">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">
_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="http://lists.osgeo.org/mailman/listinfo/gdal-dev">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
</body>
</html>