[GRASS-dev] new v.in.geonames: problems with UTF-8 Unicode text

[Markus Neteler]

On Tue, Jul 1, 2008 at 7:12 PM, Glynn Clements <glynn at gclements.plus.com> wrote:
>
> Markus Neteler wrote:
...
>> I assume that the escape magic in db/drivers/postgres/execute.c isn't
>> perfect. Any suggestions here?
>
> Is there actually a problem? All I see are warnings.
>
> If I try inserting the above into a test table using the psql utility,
> I get exactly what I would expect.
...
> Note that the backslash is present in the file, which appears to be an
> error; many other entries have literal apostrophes without the
> backslash.

Ah, I overlooked that there is misfit in the input data.

(Un)related a segfault: When copying the map from the postgresql mapset
into the sqlite mapset, it crashes:

# in SQLite mapset:
GRASS 6.4.svn (latlong_tbe_climate): > gdb g.copy
GNU gdb 6.6-3mdv2008.0 (Mandriva Linux release 2008.0)
...
This GDB was configured as "x86_64-mandriva-linux-gnu"...
Using host libthread_db library "/lib64/libthread_db.so.1".
(gdb) r vect=it at postgresql,test --o
Starting program:
/home/neteler/grass64/dist.x86_64-unknown-linux-gnu/bin/g.copy
vect=it at postgresql,test --o
[Thread debugging using libthread_db enabled]
[New Thread 46964157956272 (LWP 10388)]
warning: Lowest section in /usr/lib64/libicudata.so.36 is .gnu.hash at
0000000000000120
Copy vector <it at postgresql> to current mapset as <test>
WARNING: Vector map <test> already exists and will be overwritten
WARNING: Table <test> linked to vector map <test> does not exist

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 46964157956272 (LWP 10388)]
0x00002ab6aa9e8888 in db_execute_immediate (driver=0xab8be18d88e1a38c,
SQLstatement=0x7fff00de1240)
    at c_execute.c:16
16          db__set_protocol_fds (driver->send, driver->recv);
(gdb) bt full
#0  0x00002ab6aa9e8888 in db_execute_immediate
(driver=0xab8be18d88e1a38c, SQLstatement=0x7fff00de1240)
    at c_execute.c:16
        ret_code = 32767
#1  0x00002ab6aa9ea276 in db__copy_table (from_drvname=0x610420 "pg",
    from_dbname=0x6103f0 "host=localhost,dbname=postgis",
from_tblname=0x6103b0 "it",
    to_drvname=0x610540 "sqlite",
    to_dbname=0x610600
"/home/neteler/grassdata/latlong_tbe_climate/sqlite/sqlite.db",
    to_tblname=0x6104c0 "test", where=0xbf82e3a482e32ca2 <Address
0xbf82e3a482e32ca2 out of bounds>,
    select=0x85e5a282e3aa83e3 <Address 0x85e5a282e3aa83e3 out of bounds>,
    selcol=0x2cbd9be58c92e5b1 <Address 0x2cbd9be58c92e5b1 out of
bounds>, ivals=0x88e5a7a4e58f84e6,
    nvals=-1846924119) at copy_tab.c:321
        select = -999011943
        col = -1109350210
        ncols = -1950555982
        sqltype = -527778336
        ctype = -1109357636
        more = -533944643
        selcol_found = -1296179037
        buf = "'''Itaria,An Eadailt,An Iodail,An
Iodáil,Etalie,Itaalia,Itaalje,Itaglia,Itali,Italia,Italia (Y),Italia
(�\235),Italie,Italien,Italija,Italio,Italiya,Italië,Italja,Italska,Italujo,Italy,Italya,Ita�\202ia,I"...
        ivalues = (int *) 0xe0a8bde02c8dbce0
        from_handle = {dbName = {string = 0x610680
"host=localhost,dbname=postgis", nalloc = 30}, dbSchema = {
    string = 0x610980 "", nalloc = 1}}
        to_handle = {dbName = {string = 0x611180
"/home/neteler/grassdata/latlong_tbe_climate/sqlite/sqlite.db",
    nalloc = 61}, dbSchema = {string = 0x611980 "", nalloc = 1}}
        tblname = {string = 0x2ab6aa1293b0 "", nalloc = 0}
        sql = {
  string = 0x615b70 "insert into test values ( 14885, 3175395,
'Italian Republic', 'Italian Republic', '''Itaria,An Eadailt,An
Iodail,An Iodáil,Etalie,Itaalia,Itaalje,Itaglia,Itali,Italia,Italia
(Y),Italia (�\235),Italie,It"..., nalloc = 1277}
        value_string = {
  string = 0x6156c0 "''Itaria,An Eadailt,An Iodail,An
Iodáil,Etalie,Itaalia,Itaalje,Itaglia,Itali,Italia,Italia (-Y),Italia
(�\235),Italie,Italien,Italija,Italio,Italiya,Italië,Italja,Italska,Italujo,Italy,Italya,Ita�\202ia,It"...,
nalloc = 1192}
        tblnames = (dbString *) 0x6119a0
        table = (dbTable *) 0x8bbce08abde080be
        out_table = (dbTable *) 0xe12cb2bde0a3bde0
        cursor = {token = 0, driver = 0x6109d0, table = 0x611a10,
column_flags = 0x0, type = 1, mode = 0}
        column = (dbColumn *) 0x9083e1a283e19883
        value = (dbValue *) 0x83e19883e19a83e1
        colname = 0x8a88e1a38ce12c90 <Address 0x8a88e1a38ce12c90 out of bounds>
        from_driver = (dbDriver *) 0xe12c958ae1ab8be1
        to_driver = (dbDriver *) 0xab8be18d88e1a38c
        count = 1
        i = -509436191
#2  0x82e3aa83e3bf82e3 in ?? ()
No symbol table info available.
#3  0xbf82e3a482e32ca2 in ?? ()
No symbol table info available.
#4  0x85e5a282e3aa83e3 in ?? ()
No symbol table info available.
#5  0x2cbd9be58c92e5b1 in ?? ()
No symbol table info available.
#6  0x88e5a7a4e58f84e6 in ?? ()
No symbol table info available.
#7  0x8a84eab491ea2ca9 in ?? ()
No symbol table info available.
#8  0xedb49dec2cba86ea in ?? ()
No symbol table info available.
#9  0x8495ecaca6eb8883 in ?? ()
No symbol table info available.
#10 0x0000000000610027 in ?? ()
No symbol table info available.
#11 0x0000000000610540 in ?? ()
No symbol table info available.
#12 0x00000000006103b0 in ?? ()
No symbol table info available.
#13 0x00000000006103f0 in ?? ()
No symbol table info available.
#14 0x0000000000610420 in ?? ()
No symbol table info available.
#15 0x00007fff00de5500 in ?? ()
No symbol table info available.
#16 0x00002ab6a9efec66 in Vect_copy (in=Cannot access memory at
address 0xa482e32c61693539
) at map.c:238
        i = Cannot access memory at address 0xa482e32c61697249
(gdb)

I tried to debug this with ddd but got lost in the detached processes
(or however that was called) of th DBMI driver.

Markus