[GRASS-dev] vector libs: file based spatial index

[Benjamin Ducke]

Great work!

In keeping with good GRASS traditions, I would suggest to have this
user-configurable via an env var. Perhaps:

GRASS_SPATIAL_INDEX =
	AUTO (default: keep in mem if enough RAM available)
	FILE
	MEMORY

Cheers,

Ben


Moritz Lennert wrote:
> On 23/06/09 20:28, Markus GRASS wrote:
>> Paolo Cavallini wrote:
>>> Markus GRASS ha scritto:
>>>  
>>>> What to do now? Leave it all in memory as in grass6, build in memory
>>>> then write out (risk of running out of memory on massive datasets), or
>>>> keep it always in file? I'll not commit any time soon (also waiting for
>>>> the lib/raster commotion to settle down), I need feedback on how to
>>>> proceed or if I should forget about it.
>>>>     
>>> I think advice from Radim would be very useful here.
>>> All the best.
>>>   
>> OK, let me rephrase. I think I have two alternatives to the current
>> implementation of the vector spatial index and would like to know if
>> grass7 should get 1) faster vector display and lower memory consumption
>> at the cost of (sometimes) slower vector processing [1], 2) faster
>> vector display, a similar speed in vector processing but keep the risk
>> of running out of memory when processing large datasets, or 3) no
>> changes to the spatial index. IMHO this should be a general decision of
>> the GRASS community, not of one or two developers.
> 
> What size of vector data are we talking about concerning the risk of 
> running out of memory ? Would it be possible to implement both 1) and 2) 
> with 2) being the default and a flag to switch to 1) for very large 
> vectors ?
> 
> You wrote:
>> Considering that a file based spatial index is only useful for massive
>> vectors where memory can become a limiting factor, I hesitate to commit
>> to trunk.
> 
> Well, I don't know what you call massive, but one of the main problems 
> with the memory based index is that currently the spatial index is 
> rebuilt for each run of v.what as it is stored no where. This makes 
> querying large (e.g. 20-30.000 polygons) _very_ slow when using the GUI 
> (which will be the only option in grass7 as xmons have disappeared).
> 
> I'm not sure I understand everything correctly here, but I have the 
> feeling that there are two questions here:
> 
> 1) Should we have a file-based storage of the spatial index ? This can 
> then be read into memory when necessary, which still should be faster 
> than rebuilding it each time.
> 
> To this I clearly say +1.
> The question here is: how to make sure the index is up to date ?
> 
> 2) Should the entire treatment of the index be file-based, i.e. the 
> index is never read into memory, but always accessed via file, with the 
> speed penalties you spoke about.
> 
> To this I would say, if we can have a file-based, permanent storage of 
> the index, but read into memory for treatment, unless a flag says not to 
> load it into memory, than this would probably be the ideal, within my 
> limited understanding of the issue.
> 
> Moritz
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
> 
> 


-- 
Benjamin Ducke
Senior Geospatial Consultant

Oxford Archaeology
Janus House
Osney Mead
OX2 0ES
Oxford, U.K.

Tel: +44 (0)1865 263 800 (switchboard)
Tel: +44 (0)1865 980 758 (direct)
Fax :+44 (0)1865 793 496
benjamin.ducke at oxfordarch.co.uk




------
Files attached to this email may be in ISO 26300 format (OASIS Open Document Format). If you have difficulty opening them, please visit http://iso26300.info for more information.