[GRASS-dev] vector libs: file based spatial index

[Paul Kelly]

On Thu, 25 Jun 2009, Markus GRASS wrote:

[...]
>> very little about in this case. E.g. for completely random access
>> there might not be a lot of gain.
>
> It is completely random, the next chunk to be read/written can be
> anywhere in the file.
>
[...]
>> But if there was random access only within a certain section of a
>> file, that section could be mapped into memory and access would then
>> be quite fast.
>
> Does it make sense to always map a different section of the file? This
> mapping would then need to replace each call to fread/fwrite because of
> completely random access. I'm currently using the same method like for
> the coor file which seems to work fine so far.

No, I can't imagine that would give much improvement. The scenario I had 
hoped for was that when processing a particular sub-region of a vector 
map, all the needed data from the index would be near each other in memory 
(and thus in the file), thus that area of the file (I was thinking up to 
several hundred megabytes max. for performance reasons, but this could 
obviously scale with increased datasizes) could be mapped into memory and 
intensive I/O (or just reading) done there, with the memory-mapping 
mechanism automatically taking care of the reads and writes from/to the 
underlying file.

>From what you have said it sounds like that would need a complete redesign 
of the algorithm used for indexing, or might not be possible. The other 
option would be to map the whole of the index file into memory; it might 
be worth trying, but as you have already tried to optimise reading/writing 
performance based on knowledge of the algorithm, it is unlikely the OS 
could do any better. Still might be worth trying, but maybe too much work. 
So I think I should shut up now since (as is obvious) I don't know how the 
indexing algorithm works...

Paul