[GRASS5] Re: v.in.shp on solaris2.7,5.0.0pre3

[David D Gray]

Markus Neteler wrote:

> On Fri, Feb 08, 2002 at 07:10:27PM +0000, David D Gray wrote:
> 
>>John Gillette wrote:
>>
>>
>>>Good Morning,
>>>
>>>v.in.shp only works for me on the small test files located in
>>>.../grass5.0.0pre3/src/mapdev/v.in.shp/test:
>>>  circles.shp
>>>  islets.shp
>>>  simple.shp
>>>On bigger files, it stops in the middle.
>>>
>>>
>>>
>>
>>Currently v.in.shape is a memory hog, as it builds huge monolithic 
>>structures in memory. On some systems this may be critical, and it also 
>>depends on the resources available. Can you say - what happens in the 
>>middle of an import, when it stops?
>>
>>
> 
> Hi David,
> 
> do you foresee an update to the topology engine to work segmented (in
> 5.1)?


There are two approaches to the problem of very large files. I am not 
quite sure what constitutes `average', `normal' or `professional-level' 
specs in workstation terms these days, but there is a limit, as we know, 
to the size of files that v.in.shape can import, and this limit is 
currently unrealistically severe.

1) There is the question, which really is much more general than the 
problems of import/export, of dealing with small areas at a time, in a 
way that allows optimal use of resources, but yet is transparent to the 
end-user (even the module-developer). I think this is a reasonable 
definition of the question of `segmentation'.

2) The problem of adequate use of disk-access where the problem in hand 
exceeds the resources of the available memory.

I foresee (but it may be a mirage :) ) that attacks on both of these are 
possible for GRASS 5.1. In the case of disk-access, then even for the 
stable release, the use of dbm should solve the problem of large file 
imports.


> At time I have to process a 300000 lines SHAPE file, 1GB is far from
> being sufficient for v.in.shape. I am asking you as we discussed the
> memory problem some time ago.
> 
> I have successfully plugged in the SHAPE into 5.1, but v.build 
> eats all memory. Also 5.0 processing is impossible.
> 



The techniques for segmented import would also provide a new means of 
building and maintaining topology. So, the problem of making v.build 
efficient and effective is largely the same as that of solving the 
import problems.

This will be the next development after the shape/mif import filters are 
ready, and it builds on principles already incorporated in these. 
Currently I'm concentarting on getting these ready. There will also be - 
shortly afterwards - versions for 5.1, where the main difference will be 
in importing multiple categories to the new format.

David