[GRASS-dev] new v.in.geonames: problems with UTF-8 Unicode text

[Markus Neteler]

On Wed, Jul 2, 2008 at 3:08 AM, Glynn Clements <glynn at gclements.plus.com> wrote:
> Markus Neteler wrote:
>
>> >> when using the SQLite driver, it cuts the "alternatename" field at
>> >> 256 chars due to a hardcoded limitation in describe.c. I don't
>> >> remember why not varchar() length is used there.
>> >
>> > SQLite doesn't have VARCHAR. If you create a VARCHAR column, the
>> > driver translates it to TEXT. The size you specify when the table is
>> > created isn't stored anywhere.
>>
>> This is not clear to me:
>> - I run v.in.ascii and define that I want varchar(4000).
>> - this is sent to DBMI which calls the SQLite driver
>> - in the SQLite driver 255 is hardcoded for TEXT and the
>>   "alternatename" field is cut
>> - hardcoding something much longer isn't convenient due to
>>   troubles when moving table to other DBMS
>>
>> so far understood.
>>
>> -> question: any chance to remove the hardcoded 255 limit
>>    and use instead the length I defined in the v.in.ascii call?
>
> Not easily.

It seems that an environment variables is needed in the driver which
can be used by v.in.geonames then, because:

v.in.geonames /tmp/IT.txt out=it2 --o
Converted 42860 place names.
Scanning input for column types...
Maximum input row length: 1309
Maximum number of columns: 19
Minimum number of columns: 19
WARNING: Table <it2> linked to vector map <it2> does not exist
ERROR: Length of column 4 <alternatename> (1000) is less than maximum value
       length (1190)

...
> In theory, the driver could scan the entire table and determine the
> widest element. But that would be slow for large tables, and it could
> also confuse clients trying to add data to an existing table.

Right. It would be perfect to have a "bundled" env var like:
GRASS_SQLITE_TEXTCOL_LENGTH=name,length

Or better two of them? Like this we could specify it individually for a
column and v.in.ascii would not fail.

Markus