[GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless

[Hamish]

> > for moderate and large data sets, but not for XXL.
> 
> What's XXL?

t-shirt speak for extra-extra-large.

> I need to reproject a detailed 5m DEM of one national park
> only. To do it properly, it has to be transformed into vector points,
> these will be reprojected, then a DEM in new projection will be
> reinterpolated. Unfortunatelly reprojecting a DEM as raster yields
> distrotions AFAIK.

give it a try, then compare. It might not be that bad in practice. For a
non-categorical map, try 'r.proj method=cubic' for a better result.
Randomly select test points (v.random -> v.what.rast; v.proj ->
v.what.rast) for a simple quantitative comparison or error.

> > If it is aknowledged, can we expect it to be fixed? When - soon,
> > month time, year time? Or is it going to be a "feature" and left as
> > is?

the official Free software answer to this FAQ: "It will be done when it
is ready, sooner if you help." (c.1992 ref Linux 1.0)

> And I didn't report the bug in s.surf.rst because Grass 5 is no longer
> mantained by the core dev crew.

severe bugs will be given a cursory look, data corruption bugs will be
explored, and if reported to the bug tracker (in the correct category;
5.4 is still there) any normal [unfixed] bugs listed will at minimum
give others seeing the same bug a sense of solidarity and closure. Minor
bugs will be likely ignored, unless trivial and/or a patch is supplied.
i.e. GRASS 5.4 is still maintained, just not actively.

> > This is not a bug,
> That's a very tollerant approach toward bug definition.

This is a known limitation in the data model. There is no error in the
code, it just wasn't designed to deal with that large of a dataset.
The GRASS 6 vector model was designed before the popularization of LIDAR
and multibeam sonar. Anything over 3 million points or so will use up
all your memory. Your OS should see this and kill the offending
application when this happens. All solutions and work-arounds suggested
to date have not been satisfactory, so the issue remains open until we
can figure out something better. Who knows when that will be? As far as
I understand from Radim's explainations, there appears to be no quick
fix.

An outstanding issue which can be resolved is a port of s.cellstats to
GRASS 6. This would help ameliorate the problem for some people..
Or a preprossor utility...

> >  - I don't think that a non-root user 
> > on a sensible OS can freeze the system so that a hard shutdown (pull
> > power) is required) is unfortunate, it is usually caused by 100% CPU
> > use  and swapping caused by memory being fully occupied.
>
> That was the case. like I described it - both 1GB ram and 1GB swap
> where used, total hang, even mouse pointer freezed, had to reset.

The point being that your OS should have killed the process that was 
causing the problem. GRASS just asks the OS for more memory. If the OS
keeps giving a program more memory after it has already run out and is
gasping for air, we can't help that.

> That's your perception. From my point of view Grass vector model is
> not well written yet.

It is beautifully written to do what it was intended to do. The problem
is that you (and others) want it to do something that it was not
intended to do. It can still call itself a good GIS vector engine in
good conscience, but it is not all things.

Also, I would suggest that this sort of comment is not the best way to
motivate a designer to make changes for you. Remember that nothing kills
the volunteer spirit faster than demands and insulting the creator's
baby.

If this problem affects enough people (and developers) it will
eventually affect someone who can a) fix it or b) pay someone else to
fix it; and then it will be fixed. This problem seems to affect a number
of people, so there is hope. Can't promise you more than that.
 

Trevor:
> First, I searched the archives for the old archives and was unable to
> find any references to r.to.vect halting the system.

No, you won't find it looking for r.to.vect; the same issue was mostly
discussed with respect to creating vector points with v.in.ascii. Try as
search terms: "LIDAR" "memory" "v.in.ascii" "valgrind"; for emails from
Radim, Helena, and myself.

> Second, if this is a problem that has come up repeatedly, this is an
> indication that work is needed in this area.

I think we all agree that is true. But we don't know of a good solution
yet.

> Probably updating the documentation outlining this problem would be a
> good idea

Yes, it would be good to mention in the v.in.ascii and r.to.vect help
pages that creating several million data points may use large amounts of
memory, which may cause problems.

Note that once the import issue is solved, other modules start to have
problems with these huge datasets too.


> Third, as a former database programmer, I would say that when a
> program fails, it should fail early, loudly (with abundant information
> to help solve the problem), and nicely (not halting the system).
> AFAICT Maciek was using r.to.vect as the program was intended,
> converting a raster to a vector. If there are is some sort of
> artificial file size issues, that should be clearly stated in the
> documentation, or better yet the program should check is first and
> fail without wasting the users time or computing resources. Based on
> these reasonable user and developer based criteria, this is a bug.
> That said I don't know what the issues are surrounding this process,
> but I took a brief look at the vector specification in the developers
> documentation this afternoon out of curiosity and I can't see how this
> conversion would need to be written for point files where file size or
> even computing resources should be an issue.

Band-aid approach (curing the symptom, not the cause):
Once number of points is known, a calculation of memory use could be
done (~300 bytes per vector point?, best create a test point & sizeof()
rather than hardcode "300"). G_malloc() and G_free() could be
called as a test, which will call G_fatal_error() if the dataset is too
huge to complete. I don't think this reduces the need for a solution,
just makes the failure friendlier.

Another angle of attack is to get that 300 bytes per point number lower.


10 million points should be possible given enough swap space, the
current model, and today's hardware; 50 million is really pushing it.