[GRASS-dev] Re: About the vector changes

[Markus Metz]

Glynn Clements wrote:
> Markus Metz wrote:
>
>   
>> Yes, if you want to work with vectors not in the current mapset using 
>> grass7, you have to rebuild topology for all vectors in the 
>> corresponding mapset first.
>>     
>
> So it's intentional that GRASS7 is no longer compatible with previous
> vector maps?
>   
Yes, for two reasons. GRASS7 expects now a sidx file while GRASS6 
doesn't, and the last GRASS version writing a sidx file (I think 
GRASS57) wrote it differently to the new format. I have now changed the 
format numbers and set forward/backward compatibility info. GRASS6 can 
open GRASS7 vectors, but GRASS7 can't open GRASS6 or GRASS5 vectors, 
rebuilding topology is required. IOW, I broke backwards compatibility 
which is against GRASS development policy I guess. My argument is that 
enabling backwards compatibility would be about the same effort like 
rebuilding topology. I could write a module that converts any existing 
sidx file to the new format or builds the spatial index from topo if no 
sidx file is found, then writes a new sidx file in the new format. That 
would avoid rebuilding the core topology and category index.

The main change and justification is that GRASS7 can search the spatial 
index in file without loading the spatial index to memory. Only the 
header is loaded. The last GRASS version that wrote a sidx file could 
not search in file, the nodes were loaded from file and the spatial 
index was recreated in memory first. The sidx file was missing the 
location of children for each node of the RTree, that was not fixable 
with the previous method because it was depth-first pre-order traversal. 
I use now depth-first post-order traversal and thus know the location of 
all children once I write out a node. This is needed to do search and 
query in file. All functions for reading and writing the sidx file had 
to be rewritten from scratch. Further on, the previous RTree 
implementation was based on a testing version written by the original 
author Tony Guttman and was missing some essentials that belong to a 
standard data structure (ideas for modification taken from dglib and 
libavl). The new R*Tree used in GRASS7 has a new interface and new 
algorithms. All changes affect only the vector libs, the vector API for 
writing modules remained unchanged. I guess I should have mentioned that 
earlier.
> That isn't necessarily a problem, but I must have missed the
> announcement. 
Was hidden here:
http://lists.osgeo.org/pipermail/grass-dev/2009-July/045098.html
> But the error message needs to be clear; and accurate:
> the problem isn't a lack of LFS support, and the map *wan't* created
> with the LFS enabled (the map hasn't changed in 5 years).
>   
Yes, it has changed, at least the spearfish dataset for GRASS6. Files of 
vector fields were last modified on April 11 2005 and Jan 17 2006.

-rw-r--r-- 1 mmetz users 4117 2006-01-17 17:50 cidx
-rw-r--r-- 1 mmetz users 27905 2005-04-11 14:26 coor
-rw-r--r-- 1 mmetz users 55 2005-04-11 14:26 dbln
-rw-r--r-- 1 mmetz users 278 2005-04-11 14:26 head
-rw-r--r-- 1 mmetz users 191 2005-04-11 14:26 hist
-rw-r--r-- 1 mmetz users 25161 2006-01-17 17:50 topo

Downloaded today 2009-08-08.

>   
>>> FWIW, the complete structure is:
>>>       
>
> [snip]
>
>   
>>> Other than off_t_size = 42, ptr->spidx_port is all zeros.
>>>       
>> Another problem is ptr->port.off_t_size = 0, it should be 4, as 
>> ptr->sidx_port.off_t_size.
>> topo was read successfully, reading sidx failed, cidx was not yet read. 
>> The libraries should not even attempt to read sidx if that file doesn't 
>> exist. The same test is done for all vector files. I don't understand 
>> why that happens, I get the correct error messages.
>>     
>
> PERMANENT/vector/fields contains:
>
> 	-rw-r--r-- 1 root root  4117 May 25  2004 cidx
> 	-rw-r--r-- 1 root root 27905 May 25  2004 coor
> 	-rw-r--r-- 1 root root    55 May 25  2004 dbln
> 	-rw-r--r-- 1 root root   278 May 25  2004 head
> 	-rw-r--r-- 1 root root   186 May 25  2004 hist
> 	-rw-r--r-- 1 root root 24538 May 25  2004 sidx
>   
There is no sidx file for vectors in
http://grass.osgeo.org/sampledata/spearfish_grass60data-0.3.tar.gz
> 	-rw-r--r-- 1 root root 25161 May 25  2004 topo
>
>
> The files are unmodified from:
>
> 	-rw-r--r-- 1 glynn root 19965571 Oct 16  2004 spearfish_grass57data.tar.gz
>
> 	c7a356fcc9ab649f73973315cba205cb  spearfish_grass57data.tar.gz
>   
Ah, ok, you use the grass57 dataset. I have now changed the version 
number and compatibility info, you should now get reasonable 
warnings/errors with the spearfish57 dataset.
>
> Offsets need to be calculated as off_t, not int or long. Casting them
> when lseek/G_fseek are called is too late. IOW:
>
> 	int row, cols, cols, bytes_per_cell;
> 	off_t offset = row * (col * rows + col) * bytes_per_cell;
>
> needs to be e.g.:
>
> 	off_t offset = ((off_t) row * cols + col) * bytes_per_cell;
>
> so that off_t is used throughout the calculation.
>   
OK.
> Also, this needs to be checked for anything which uses VECT_CFLAGS,
> which isn't limited to v.* modules (a few g.* and r.* modules use
> vector functions).
>   
I fixed r.drain, v.in.dxf, v.vol.rst. AFAICT all other g.*, r.* and v.* 
modules are ok. Please let me know if I missed something.

I got carried away (replacing all fseek/ftell occurrences I could find 
with G_fseek/G_ftell, adjusting off_t as you showed above) and also made 
r3.in|out.v5d LFS-safe, but did not submit. Should I?

Markus M