[GRASS-dev] Re: About the vector changes

[Glynn Clements]

Markus Metz wrote:

> > Vector maps no longer work:
> >
> > $ v.info fields
> > ERROR: Spatial index was written with LFS but this GRASS version does not
> >        support LFS. Try to rebuild topology or upgrade GRASS.
> >   
> Can't reproduce. This was fields from spearfish I assume.

Yes.

> Without building topo I get
> 
> ERROR: Unable to open vector map <fields at PERMANENT> on level 2. Try to 
> rebuild vector topology by v.build.
> 
> So I do
> v.build map=fields
> v.info fields now produces standard output

Running v.build requires that you own the mapset which contains it.

g.copy produces the same error, but it does appear to copy the map.
Running v.build on the copy results in a working map.

> > BTW, spidx_port.off_t_size is 42, so the issue is that the code
> > expects the new format, not that sizeof(off_t) is wrong.
> 
> Weird, spidx_port.off_t_size should be 4 or 8, and 8 is only used when 
> necessary, same like for reading/writing topo. I haven't included some 
> debug info on spidx_port.off_t_size, so I guess you have added that 
> yourself? Maybe a good idea, it's already there in dig_Rd_Plus_head() 
> for reading topo. Can you also report sizeof(off_t)? That could help me 
> tracking down the bug.

sizeof(off_t) is 8.

FWIW, the complete structure is:

Breakpoint 1, G_fatal_error (
    msg=0xb7f38128 "Spatial index was written with LFS but this GRASS version does not support LFS. Try to rebuild topology or upgrade GRASS.")
    at error.c:151
151	    va_start(ap, msg);
> where
#0  G_fatal_error (
    msg=0xb7f38128 "Spatial index was written with LFS but this GRASS version does not support LFS. Try to rebuild topology or upgrade GRASS.")
    at error.c:151
#1  0xb7f332d6 in dig_Rd_spidx_head (fp=0xbfdb6920, ptr=0xbfdb672c)
    at spindex_rw.c:280
#2  0xb7fd93e2 in Vect_open_sidx (Map=0xbfdb6720, mode=0) at open.c:854
#3  0xb7fd7f99 in Vect__open_old (Map=0xbfdb6720, name=0x8e827a0 "fields", 
    mapset=0x804af50 "", update=0, head_only=1) at open.c:262
#4  0xb7fd87a8 in Vect_open_old_head (Map=0xbfdb6720, 
    name=0x8e827a0 "fields", mapset=0x804af50 "") at open.c:501
#5  0x08049860 in main (argc=3, argv=0xbfdb6cf4) at main.c:104
> frame 1
#1  0xb7f332d6 in dig_Rd_spidx_head (fp=0xbfdb6920, ptr=0xbfdb672c)
    at spindex_rw.c:280
280		G_fatal_error("Spatial index was written with LFS but this "
> print *ptr
$1 = {Version_Major = 5, Version_Minor = 0, Back_Major = 5, Back_Minor = 0, 
  spidx_Version_Major = 5, spidx_Version_Minor = 0, spidx_Back_Major = 5, 
  spidx_Back_Minor = 0, cidx_Version_Major = 0, cidx_Version_Minor = 0, 
  cidx_Back_Major = 0, cidx_Back_Minor = 0, with_z = 0, spidx_with_z = 0, 
  off_t_size = 4, head_size = 142, spidx_head_size = 0, cidx_head_size = 0, 
  release_support = 0, port = {byte_order = 0, off_t_size = 0, 
    dbl_cnvrt = "\000\001\002\003\004\005\006\a", 
    flt_cnvrt = "\000\001\002\003", lng_cnvrt = "\000\001\002\003", 
    int_cnvrt = "\000\001\002\003", shrt_cnvrt = "\000\001", 
    off_t_cnvrt = "\000\001\002\003\004\005\006\a", dbl_quick = 1, 
    flt_quick = 1, lng_quick = 1, int_quick = 1, shrt_quick = 1, 
    off_t_quick = 1}, spidx_port = {byte_order = 0, off_t_size = 42, 
    dbl_cnvrt = "\000\000\000\000\000\000\000", flt_cnvrt = "\000\000\000", 
    lng_cnvrt = "\000\000\000", int_cnvrt = "\000\000\000", 
    shrt_cnvrt = "\000", off_t_cnvrt = "\000\000\000\000\000\000\000", 
    dbl_quick = 0, flt_quick = 0, lng_quick = 0, int_quick = 0, 
    shrt_quick = 0, off_t_quick = 0}, cidx_port = {byte_order = 0, 
    off_t_size = 0, dbl_cnvrt = "\000\000\000\000\000\000\000", 
    flt_cnvrt = "\000\000\000", lng_cnvrt = "\000\000\000", 
    int_cnvrt = "\000\000\000", shrt_cnvrt = "\000", 
    off_t_cnvrt = "\000\000\000\000\000\000\000", dbl_quick = 0, 
    flt_quick = 0, lng_quick = 0, int_quick = 0, shrt_quick = 0, 
    off_t_quick = 0}, mode = 0, built = 0, box = {N = 4928110, S = 4913925, 
    E = 609544.37, W = 589438.87, T = 0, B = 0}, Node = 0x0, Line = 0x0, 
  Area = 0x0, Isle = 0x0, n_nodes = 217, n_edges = 0, n_lines = 271, 
  n_areas = 65, n_isles = 11, n_faces = 0, n_volumes = 0, n_holes = 0, 
  n_plines = 0, n_llines = 0, n_blines = 208, n_clines = 63, n_flines = 0, 
  n_klines = 0, n_vfaces = 0, n_hfaces = 0, alloc_nodes = 0, alloc_edges = 0, 
  alloc_lines = 0, alloc_areas = 0, alloc_isles = 0, alloc_faces = 0, 
  alloc_volumes = 0, alloc_holes = 0, Node_offset = 142, Edge_offset = 0, 
  Line_offset = 8314, Area_offset = 20157, Isle_offset = 24417, 
  Volume_offset = 0, Hole_offset = 0, Spidx_built = 0, Spidx_new = 0, 
  spidx_fp = {file = 0x8e830f0, start = 0x0, current = 0x0, end = 0x0, 
    size = 0, alloc = 0, loaded = 0}, spidx_node_fname = 0x0, 
  Node_spidx_offset = 0, Line_spidx_offset = 0, Area_spidx_offset = 0, 
  Isle_spidx_offset = 0, Face_spidx_offset = 0, Volume_spidx_offset = 0, 
  Hole_spidx_offset = 0, Node_spidx = 0x8e834e8, Line_spidx = 0x8e83740, 
  Area_spidx = 0x8e83998, Isle_spidx = 0x8e83bf0, Face_spidx = 0x0, 
  Volume_spidx = 0x0, Hole_spidx = 0x0, update_cidx = 0, n_cidx = 0, 
  a_cidx = 5, cidx = 0x8e83328, cidx_up_to_date = 0, coor_size = 27905, 
  coor_mtime = 0, do_uplist = 0, uplines = 0x0, alloc_uplines = 0, 
  n_uplines = 0, upnodes = 0x0, alloc_upnodes = 0, n_upnodes = 0}

Other than off_t_size = 42, ptr->spidx_port is all zeros.

> > Also, lib/vector/diglib doesn't actually enable LFS; the Makefile
> > needs:
> >
> > ifneq ($(USE_LARGEFILES),)
> > 	EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
> > endif
> >   
> No, because this is set system-wide in Grass.make
> https://trac.osgeo.org/grass/browser/grass/trunk/include/Make/Grass.make#L82

Okay; I missed that part.

That's likely to cause problems for anything which uses $(VECT_CFLAGS)
but which isn't itself 64-bit aware (i.e. uses fseek/ftell, calculates
offsets as int/long).

> because not only the library but also all modules including vector.h 
> must have 64bit enabled.

That means all such modules must be made 64-bit safe. Without
_FILE_OFFSET_BITS=64, open(), fopen() etc will simply refuse to open
files larger than 2GiB. If you set _FILE_OFFSET_BITS=64 in code which
wasn't designed for it, ftell() will fail once you get past the 2GiB
mark, storing the result of lseek() in a long will silently truncate
it, etc.

This change has the potential to corrupt files, so it needs to be
completed ASAP.

> Instead of adding the appropriate lines to each 
> single Makefile I decided to set it system-wide, that also avoids 
> problems with addon modules. It worked so far on Linux 32bit and Linux 
> 64bit. What platform are you using?

Linux/x86 (32-bit).

> I hesitate to ask, but have you made funny changes to your local
> copy?

No.

-- 
Glynn Clements <glynn at gclements.plus.com>