[GRASS-dev] Re: About the vector changes

[Markus Metz]

Glynn Clements wrote:
> Markus Metz wrote:
>
>   
>> Without building topo I get
>>
>> ERROR: Unable to open vector map <fields at PERMANENT> on level 2. Try to 
>> rebuild vector topology by v.build.
>>
>> So I do
>> v.build map=fields
>> v.info fields now produces standard output
>>     
>
> Running v.build requires that you own the mapset which contains it.
>   
Yes, if you want to work with vectors not in the current mapset using 
grass7, you have to rebuild topology for all vectors in the 
corresponding mapset first.
> g.copy produces the same error, but it does appear to copy the map.
> Running v.build on the copy results in a working map.
>   
Not here, g.copy also copies vector maps where one or more support files 
(topo, sidx, cidx) is missing. This is correct because g.copy should 
also copy level 1 vectors that don't have these support files.
>   
>>> BTW, spidx_port.off_t_size is 42, so the issue is that the code
>>> expects the new format, not that sizeof(off_t) is wrong.
>>>       
>> Weird, spidx_port.off_t_size should be 4 or 8, and 8 is only used when 
>> necessary, same like for reading/writing topo. I haven't included some 
>> debug info on spidx_port.off_t_size, so I guess you have added that 
>> yourself? Maybe a good idea, it's already there in dig_Rd_Plus_head() 
>> for reading topo. Can you also report sizeof(off_t)? That could help me 
>> tracking down the bug.
>>     
>
> sizeof(off_t) is 8.
>
> FWIW, the complete structure is:
>
> Breakpoint 1, G_fatal_error (
>     msg=0xb7f38128 "Spatial index was written with LFS but this GRASS version does not support LFS. Try to rebuild topology or upgrade GRASS.")
>     at error.c:151
> 151	    va_start(ap, msg);
>   
>> where
>>     
> #0  G_fatal_error (
>     msg=0xb7f38128 "Spatial index was written with LFS but this GRASS version does not support LFS. Try to rebuild topology or upgrade GRASS.")
>     at error.c:151
> #1  0xb7f332d6 in dig_Rd_spidx_head (fp=0xbfdb6920, ptr=0xbfdb672c)
>     at spindex_rw.c:280
> #2  0xb7fd93e2 in Vect_open_sidx (Map=0xbfdb6720, mode=0) at open.c:854
> #3  0xb7fd7f99 in Vect__open_old (Map=0xbfdb6720, name=0x8e827a0 "fields", 
>     mapset=0x804af50 "", update=0, head_only=1) at open.c:262
> #4  0xb7fd87a8 in Vect_open_old_head (Map=0xbfdb6720, 
>     name=0x8e827a0 "fields", mapset=0x804af50 "") at open.c:501
> #5  0x08049860 in main (argc=3, argv=0xbfdb6cf4) at main.c:104
>   
>> frame 1
>>     
> #1  0xb7f332d6 in dig_Rd_spidx_head (fp=0xbfdb6920, ptr=0xbfdb672c)
>     at spindex_rw.c:280
> 280		G_fatal_error("Spatial index was written with LFS but this "
>   
>> print *ptr
>>     
> $1 = {Version_Major = 5, Version_Minor = 0, Back_Major = 5, Back_Minor = 0, 
>   spidx_Version_Major = 5, spidx_Version_Minor = 0, spidx_Back_Major = 5, 
>   spidx_Back_Minor = 0, cidx_Version_Major = 0, cidx_Version_Minor = 0, 
>   cidx_Back_Major = 0, cidx_Back_Minor = 0, with_z = 0, spidx_with_z = 0, 
>   off_t_size = 4, head_size = 142, spidx_head_size = 0, cidx_head_size = 0, 
>   release_support = 0, port = {byte_order = 0, off_t_size = 0, 
>     dbl_cnvrt = "\000\001\002\003\004\005\006\a", 
>     flt_cnvrt = "\000\001\002\003", lng_cnvrt = "\000\001\002\003", 
>     int_cnvrt = "\000\001\002\003", shrt_cnvrt = "\000\001", 
>     off_t_cnvrt = "\000\001\002\003\004\005\006\a", dbl_quick = 1, 
>     flt_quick = 1, lng_quick = 1, int_quick = 1, shrt_quick = 1, 
>     off_t_quick = 1}, spidx_port = {byte_order = 0, off_t_size = 42, 
>     dbl_cnvrt = "\000\000\000\000\000\000\000", flt_cnvrt = "\000\000\000", 
>     lng_cnvrt = "\000\000\000", int_cnvrt = "\000\000\000", 
>     shrt_cnvrt = "\000", off_t_cnvrt = "\000\000\000\000\000\000\000", 
>     dbl_quick = 0, flt_quick = 0, lng_quick = 0, int_quick = 0, 
>     shrt_quick = 0, off_t_quick = 0}, cidx_port = {byte_order = 0, 
>     off_t_size = 0, dbl_cnvrt = "\000\000\000\000\000\000\000", 
>     flt_cnvrt = "\000\000\000", lng_cnvrt = "\000\000\000", 
>     int_cnvrt = "\000\000\000", shrt_cnvrt = "\000", 
>     off_t_cnvrt = "\000\000\000\000\000\000\000", dbl_quick = 0, 
>     flt_quick = 0, lng_quick = 0, int_quick = 0, shrt_quick = 0, 
>     off_t_quick = 0}, mode = 0, built = 0, box = {N = 4928110, S = 4913925, 
>     E = 609544.37, W = 589438.87, T = 0, B = 0}, Node = 0x0, Line = 0x0, 
>   Area = 0x0, Isle = 0x0, n_nodes = 217, n_edges = 0, n_lines = 271, 
>   n_areas = 65, n_isles = 11, n_faces = 0, n_volumes = 0, n_holes = 0, 
>   n_plines = 0, n_llines = 0, n_blines = 208, n_clines = 63, n_flines = 0, 
>   n_klines = 0, n_vfaces = 0, n_hfaces = 0, alloc_nodes = 0, alloc_edges = 0, 
>   alloc_lines = 0, alloc_areas = 0, alloc_isles = 0, alloc_faces = 0, 
>   alloc_volumes = 0, alloc_holes = 0, Node_offset = 142, Edge_offset = 0, 
>   Line_offset = 8314, Area_offset = 20157, Isle_offset = 24417, 
>   Volume_offset = 0, Hole_offset = 0, Spidx_built = 0, Spidx_new = 0, 
>   spidx_fp = {file = 0x8e830f0, start = 0x0, current = 0x0, end = 0x0, 
>     size = 0, alloc = 0, loaded = 0}, spidx_node_fname = 0x0, 
>   Node_spidx_offset = 0, Line_spidx_offset = 0, Area_spidx_offset = 0, 
>   Isle_spidx_offset = 0, Face_spidx_offset = 0, Volume_spidx_offset = 0, 
>   Hole_spidx_offset = 0, Node_spidx = 0x8e834e8, Line_spidx = 0x8e83740, 
>   Area_spidx = 0x8e83998, Isle_spidx = 0x8e83bf0, Face_spidx = 0x0, 
>   Volume_spidx = 0x0, Hole_spidx = 0x0, update_cidx = 0, n_cidx = 0, 
>   a_cidx = 5, cidx = 0x8e83328, cidx_up_to_date = 0, coor_size = 27905, 
>   coor_mtime = 0, do_uplist = 0, uplines = 0x0, alloc_uplines = 0, 
>   n_uplines = 0, upnodes = 0x0, alloc_upnodes = 0, n_upnodes = 0}
>
> Other than off_t_size = 42, ptr->spidx_port is all zeros.
>   
Another problem is ptr->port.off_t_size = 0, it should be 4, as 
ptr->sidx_port.off_t_size.
topo was read successfully, reading sidx failed, cidx was not yet read. 
The libraries should not even attempt to read sidx if that file doesn't 
exist. The same test is done for all vector files. I don't understand 
why that happens, I get the correct error messages.
>   
>> No, because this is set system-wide in Grass.make
>> https://trac.osgeo.org/grass/browser/grass/trunk/include/Make/Grass.make#L82
>>     
>
>
> That's likely to cause problems for anything which uses $(VECT_CFLAGS)
> but which isn't itself 64-bit aware (i.e. uses fseek/ftell, calculates
> offsets as int/long).
>
>   
>> because not only the library but also all modules including vector.h 
>> must have 64bit enabled.
>>     
>
> That means all such modules must be made 64-bit safe. Without
> _FILE_OFFSET_BITS=64, open(), fopen() etc will simply refuse to open
> files larger than 2GiB. If you set _FILE_OFFSET_BITS=64 in code which
> wasn't designed for it, ftell() will fail once you get past the 2GiB
> mark, storing the result of lseek() in a long will silently truncate
> it, etc.
>
> This change has the potential to corrupt files, so it needs to be
> completed ASAP.
>   
OK. So I have to check and correct all occurrences of ftell and fseek in 
all vector modules and make sure the result of lseek is stored in off_t 
not long, and that offset passed to lseek is of type off_t not long. 
What else needs to be done?

open() and fopen() should behave the same like in the vector libs 
because of VECT_CFLAGS
>   
>> Instead of adding the appropriate lines to each 
>> single Makefile I decided to set it system-wide, that also avoids 
>> problems with addon modules. It worked so far on Linux 32bit and Linux 
>> 64bit. What platform are you using?
>>     
>
> Linux/x86 (32-bit).
>   
With or without LFS?

I'm going to test (again) on Linux 32bit with and without LFS, it will 
take some time (a day or two).

Markus M