[GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless

[Maciek Sieczka]

On pon, 2005-12-05 at 19:11 +0100, Roger Bivand wrote:
> On Mon, 5 Dec 2005, Maciek Sieczka wrote:
> 
> > On pon, 2005-12-05 at 13:38 +0100, Radim Blazek via RT wrote:
> > > please read old mails on this problem. I dont have time to explain it
> > > again and again. AFAIK there are no big memory leaks.
> > 
> > Is it aknowledged by Grass developers that a machine freeze at 5 mln
> > vector points file is a BUG (no matter what the reason is)?
> > 
> > If it is aknowledged, can we expect it to be fixed? When - soon, month
> > time, year time? Or is it going to be a "feature" and left as is?
> 
> I think this is unfair.

No. This is a question. I've got a task to accomplish which I can't in
Grass6 currently. I'm asking what are the chances my problem will be
fixed soon/ever. I have about 3 months or so for my task. I need to know
where I'm standing instead of "I don't have time to explain". And how do
I even know if those "old mails" reflect the current state?

It's Radim's answer that is unfair, not my reply. Or maybe I'm doing
something rude reporting bugs and I get what I deserve?

> There has been progress in GRASS 6 on this, and 
> the vector architecture is much stronger than it was in GRASS 5

Am I saying it isn't progressing? I'm saying it still not good enough.
But I'm happy with any single improvement that takes place. Sorry if I
don't express it enough. Sure it is easier to point out errors than good
things. But anyway, this are a devel list and bugtracker - a place for
discussing problems mainly.

> for moderate and large data sets, but not for XXL.

What's XXL? I need to reproject a detailed 5m DEM of one national park
only. To do it properly, it has to be transformed into vector points,
these will be reprojected, then a DEM in new projection will be
reinterpolated. Unfortunatelly reprojecting a DEM as raster yields
distrotions AFAIK.

> Have you considered using GRASS 5, which has sites, a very much simpler 
> data model for points?

I had had bad experience with vector point in Grass 6 before, so
actually it was the first thing to try sites in 5.4. Although I managed
to transform my 50 mln cells DEM into sites and reproject those,
s.surf.rst crashed on such dataset. Since it was an 8GB P4 with plenty
of swap, I didn't even try it at home on my 1GB RAM machine - I wonder
if I could get at least that far.

And I didn't report the bug in s.surf.rst because Grass 5 is no longer
mantained by the core dev crew.

Then I tried 6.1, wondering how far I cold go and hoping that when I
enconter problems, I'm more likely to be helped, ie. the bug would be
fixed. Or that at least my experience and the bug report will be somehow
appreciated. That was silly I see.

>  Have you considered tiling your data - reading 
> portions of your data and patching the resulting spline surfaces?

I would like to avoid it. Is it a good idea to mosaick DEM? Won't there
be artifacts at the connetcions?

>  Once you 
> have the surface, you can transfer it to GRASS 6, because as yet the 
> raster storage data model is effectively unchanged. 
> 
> This is not a bug,

That's a very tollerant approach toward bug definition.

>  it is a mis-match of data models and intentions.

Do you mean that the fact Grass is not able to handle even 5 mln vector
points (1 tenth of my whole possible dataset) is something normal?
What's 5 mln points? 2236x2236 points, 10x500 km GPS tracks at 1m
interval. Something a serious GIS vector model should handle perfect.

>  While 
> accepting that freezing (meaning causing total OS failure, or rather 
> occupation of all machine resources?

The latter.

>  - I don't think that a non-root user 
> on a sensible OS can freeze the system so that a hard shutdown (pull 
> power) is required) is unfortunate, it is usually caused by 100% CPU use 
> and swapping caused by memory being fully occupied.

That was the case. like I described it - both 1GB ram and 1GB swap where
used, total hang, even mouse pointer freezed, had to reset.

>  In well-written 
> software, like GRASS 6 vector

That's your perception. From my point of view Grass vector model is not
well written yet. Of course, one might say "send us a patch", and
"limited man power". Unfortunatelly I'm not able to send a patch. All I
can do for Grass is to test the code, report bugs, help in the
bugtracker a bit, help other users when I have some time (it is cool to
show off a little, isn't it). Which I do. Not as much as I would like
to, but anyway, I do contribute a bit. And even if I didn't, I do
deserve a decent reply.


> or R, say, there is a balance between how 
> things are written, perceived needs, and user perceptions. The GRASS 6 
> vector data model handles areas and lines pretty well, but because it is 
> trying harder on these, is not well suited to XXL points data sets.

Again, is 5 mln points an XXL? If it was the raster engine in Grass to
fail on 5 mln cells, would you still say it is XXL? Or that it was a
serious bug in the raster engine?


>  My 
> understanding is that the authors of the *.rst programs themselves also 
> use GRASS 5, among other things because its sites data model is very 
> simple.

Like I said, I managed to transform to sites and reprojected them on an
8GB beast but s.surf.rst crashed anyway. Can't recall the error message,
assumed it was pointless anyway since Grass 5 is no longer mantained
AAMOF. But I could retry to reproduce the error if it would mnake sense
(ie. if any chances it would be fixed in 5.4).

> 
> Best wishes,

Thanks for your interest.

Best regards,
Maciek


--------------------
W polskim Internecie s± setki milionów stron. My przekazujemy Tobie tylko najlepsze z nich!
http://katalog.epf.pl/