[Qgis-developer] Vector attribute types compatibility

[Radim Blazek]

On Tue, Jul 9, 2013 at 5:23 PM, Matthias Kuhn <matthias.kuhn at gmx.ch> wrote:
> What circumstances do exist, where we are not be able to determine the
> native type based upon QgsField metainformation, but without using
> typeName?

I cannot come to any case at this moment.

> I think we should focus on the development on an unambiguous type
> schema (QgsFields) and only use the typeName (with provider
> information?) as a very last resort.

Agreed.

> We could then introduce a method QgsField::canConvertTo( QgsField&
> other ) and let this method return yes / yes with constraints (as you
> described, truncate, round etc) / no.
> Right now, the QgsFields::operator== is abused for this operation, but
> it's not really accurate, like a number of bugs (FieldCalculator leaves
> you with a column full of NULL's after saving) have shown already. I
> already introduced this patch [1] to deal with the current state.
> I'm not sure if in addition to precision and length another field
> number of bits or bytes should be introduced? (I'm not familiar enought
> with the providers to answer this question, but I remember having seen
> INT8, INT16 etc. datatypes)

It would be probably better to work with a number of bytes instead of
length at least for some data types. Maybe we should define our own
enum for QgsFields data types, reflecting more SQL standard than C++
data types, because QVariant types are not sufficient. There is no
QVariant short (smallint, int2) for example. That way we could also
define the number of bytes just by the type (int2, int4, int8).

> Additionally, these fields (precision, length, bytes) should also be
> able to carry information like "not applicable"

What is "not applicable"?

> or "unlimited", so

Does not length = 0 mean unlimited?

> canConvertTo knows, if it needs to compare these fields also or compare
> them in a special way.
> E.g. String types won't need "precision"
> E.g. String types can have length unlimited

So is there a general consensus to use type/length/precision instead
of typeName everywhere?

Radim

> Regards,
> Matthias
>
> [1]
> https://github.com/qgis/Quantum-GIS/commit/0b780b0a5df8817415873ef2a55286566d53b293
>
> On Son 07 Jul 2013 12:26:55 CEST, Marco Hugentobler wrote:
>> Hi Radim
>>
>> Thanks for raising this important point.
>>
>>> but the same typeName may have different meaning
>>> for different drivers
>>
>>
>> How about passing a string that tells the origin provider string? The
>> data provider could then recognise if it comes from the same provider
>> and use the native types. If not, it should use the Qt type and
>> information from QgsField (e.g. length, precision. Probably more in
>> future? ).
>>
>> Regards,
>> Marco
>>
>>
>> Am 05.07.2013 14:42, schrieb Radim Blazek:
>>> I would like to ask how vector attribute types conversion between
>>> providers is supposed to work. Currently:
>>>
>>> QgsMemoryProvider:
>>> * mNativeTypes: int(10), double(20,5), string(255)
>>> * addAttributes() silently skips the field if type is not Int, Double
>>> or String, does not test length and precision, always returns true.
>>>
>>> QgsOgrProvider:
>>> * mNativeTypes: int(10), double(20,15), string(255), date, datetime
>>> (the types should be taken from OGR because depend on OGR driver, but
>>> I don't see any function in OGR to get supported types)
>>> * addAttributes(): if type is not supported, the field is not added,
>>> error is set and returns false, supported fields are added however.
>>> * createEmptyLayer() is using QgsVectorFileWriter which has its own
>>> field types handling (different from QgsOgrProvider::addAttributes())
>>> and calls directly OGR_L_CreateField()
>>> * convertField(): doing the same as QgsVectorFileWriter (copy pasted
>>> code) but not used at all
>>>
>>> QgsPostgresProvider:
>>> * mNativeTypes: int(0), longlong(0), double (20,20), string(255), date
>>> (there are more types defined but these are those with max
>>> lengh/precision per QVariant type)
>>> * addAttributes(): is using field typeName() to add fields, but
>>> typeName is always provider specific so it may become problem if
>>> fields come from another provider. If single field fails, no fields
>>> are added (transaction) and false is returned
>>> * createEmptyLayer(): is using convertField() (that is why d'n'd does
>>> not fail) on all fields and then addAttributes()
>>> * convertField(): sets field typeName based on type
>>>
>>> The problems I see:
>>> * impossible to take simply fields from one provider and add them to
>>> another one, for example:
>>>      * ogr -> memory: numbers are often double(24,15) or date
>>>      * ogr -> postgres: ogr is using String as typeName which fails as
>>> type in postgres, postgres mNativeTypes are too short
>>> * postgres provider addAttribute() takes typeName (which may be
>>> whatever) as type for add column
>>> * no method to convert not supported field to a supported one + get
>>> message (for example, "string cut to 255 characters" or "date
>>> converted to string"
>>>
>>> I believe that, from user point of view, if data are being written to
>>> a provider which does not supported all data types, then:
>>> * user should be warned before the data are written, getting a list of
>>> necessary type conversions and length/precision cuts
>>> * if confirmed by user, all the data should be written converted to
>>> supported types
>>> but there is no support for that in API.
>>>
>>> My suggestion:
>>> * add
>>>       virtual QgsField QgsVectorDataProvider::supportedField(field,
>>> ok, message)
>>>    which would try to convert unsupported field to a supported field
>>> and set message about conversion, it should almost never fail
>>> * all methods adding fields (createEmptyLayer, QgsVectorFileWriter)
>>> should use the same provider addAttributes()
>>> * provider addAttributes() should first check if all fields are
>>> supported using supportedField() and add nothing if at least one is
>>> unsupported (if some fields are not added successive addFeature()
>>> fails because of different number of attributes)
>>> * add support for date and datetime to memory provider and set
>>> benevolent length/precision limits (however it is not correct to set
>>> size/precision > 15 for double, that is the only way to make possible
>>> to copy data from OGR)
>>>
>>> I am not sure how QgsPostgresProvide::addAttributes() should work,
>>> because using typeName() may fail for fields from different provider
>>> while using type() (i.e. convertField()) would disable the possibility
>>> to specify explicitly postgres type by typeName. This is maybe
>>> crucial. In general, what defines field type - "type" or "typeName"?
>>> We could say that if typeName is unknown to postgres, it should use a
>>> type based on "type" but the same typeName may have different meaning
>>> for different drivers.
>>>
>>> I am talking about 2.1.
>>>
>>> Radim
>>> _______________________________________________
>>> Qgis-developer mailing list
>>> Qgis-developer at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>>
>
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer