[GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless

Tue Dec 6 06:31:02 EST 2005

Thanks to all Folks who replied! I appreciate your comments a lot! Sorry
if I insulted anyone. Sure nobody likes to be treated in an arogant and
dismissive way.

On wto, 2005-12-06 at 21:29 +1300, Hamish wrote:
> > > for moderate and large data sets, but not for XXL.
> > 
> > What's XXL?
> 
> t-shirt speak for extra-extra-large.

I was asking what Roger meant by "XXL datasets", might be I was not
explicit enough :).

We do have t-shirts in Poland. However, it is true we prefer hunting our
dinosaurs naked :).

> > I need to reproject a detailed 5m DEM of one national park
> > only. To do it properly, it has to be transformed into vector points,
> > these will be reprojected, then a DEM in new projection will be
> > reinterpolated. Unfortunatelly reprojecting a DEM as raster yields
> > distrotions AFAIK.
> 
> give it a try, then compare. It might not be that bad in practice. For a
> non-categorical map, try 'r.proj method=cubic' for a better result.
> Randomly select test points (v.random -> v.what.rast; v.proj ->
> v.what.rast) for a simple quantitative comparison or error.

Whatever resampling method there will be artifacts when reprojecting an
fp raster. For nearest nigbour my fp DEM will get a systematiic error
detectable on eg. aspect maps, for bilinear and more adavnced this
systematic error will decrease but original elavations will get
distorted. I don't want either.

> > > If it is aknowledged, can we expect it to be fixed? When - soon,
> > > month time, year time? Or is it going to be a "feature" and left as
> > > is?
> 
> the official Free software answer to this FAQ: "It will be done when it
> is ready, sooner if you help." (c.1992 ref Linux 1.0)

So I'm helping the way I can. Testing, repoting. Sorry if not enough. We
are all busy.

> > And I didn't report the bug in s.surf.rst because Grass 5 is no longer
> > mantained by the core dev crew.

> severe bugs will be given a cursory look, data corruption bugs will be
> explored, and if reported to the bug tracker (in the correct category;
> 5.4 is still there) any normal [unfixed] bugs listed will at minimum
> give others seeing the same bug a sense of solidarity and closure. Minor
> bugs will be likely ignored, unless trivial and/or a patch is supplied.
> i.e. GRASS 5.4 is still maintained, just not actively.

Ok, when I have time to reproduce the problem in 5.4, I will report.

> > > This is not a bug,
> > That's a very tollerant approach toward bug definition.
> 
> This is a known limitation in the data model. There is no error in the
> code, it just wasn't designed to deal with that large of a dataset.
> The GRASS 6 vector model was designed before the popularization of LIDAR
> and multibeam sonar. Anything over 3 million points or so will use up
> all your memory. Your OS should see this and kill the offending
> application when this happens. All solutions and work-arounds suggested
> to date have not been satisfactory, so the issue remains open until we
> can figure out something better. Who knows when that will be? As far as
> I understand from Radim's explainations, there appears to be no quick
> fix.

You are talking from the developer's point of view, myself from the
user's. We will never agree here.

> An outstanding issue which can be resolved is a port of s.cellstats to
> GRASS 6. This would help ameliorate the problem for some people..
> Or a preprossor utility...

Grass 6 was supposed to be sites free. If sites functionality gets to
Grass6 the point vector engine will be in danger of being less used,
tested and fixed. This we don't want. IMHO functional sites import, and
maybe export, should be all we need here. Vecto engine should be fixed
instaed to handle any dataset. Besides, although my problem refers to
vector points, I guess it would also appear for other geometry types
when milions of objects are involved - or am I wrong here?

> > >  - I don't think that a non-root user 
> > > on a sensible OS can freeze the system so that a hard shutdown (pull
> > > power) is required) is unfortunate, it is usually caused by 100% CPU
> > > use  and swapping caused by memory being fully occupied.
> >
> > That was the case. like I described it - both 1GB ram and 1GB swap
> > where used, total hang, even mouse pointer freezed, had to reset.
> 
> The point being that your OS should have killed the process that was 
> causing the problem. GRASS just asks the OS for more memory. If the OS
> keeps giving a program more memory after it has already run out and is
> gasping for air, we can't help that.
> > That's your perception. From my point of view Grass vector model is
> > not well written yet.
> 
> It is beautifully written to do what it was intended to do. The problem
> is that you (and others) want it to do something that it was not
> intended to do. It can still call itself a good GIS vector engine in
> good conscience, but it is not all things.
> 
> Also, I would suggest that this sort of comment is not the best way to
> motivate a designer to make changes for you. Remember that nothing kills
> the volunteer spirit faster than demands and insulting the creator's
> baby.

Nothing kills the volunteer user spirit for testing and reporting as his
effort not being taken into account. Most Grass developers know it and
appreciate bug reports, even if sometimes silly or wrong. Look at it his
way: Lots of people use Grass. All users must have problems. How many
report bugs in a reasonable way presently, ecluding developers? 10? Out
of hundreds or thousands using Grass? Can Grass afford loosing these 10?

> If this problem affects enough people (and developers) it will
> eventually affect someone who can a) fix it or b) pay someone else to
> fix it; and then it will be fixed. This problem seems to affect a number
> of people, so there is hope. Can't promise you more than that.

Thanks.

> Trevor:
> > First, I searched the archives for the old archives and was unable to
> > find any references to r.to.vect halting the system.
> 
> No, you won't find it looking for r.to.vect; the same issue was mostly
> discussed with respect to creating vector points with v.in.ascii. Try as
> search terms: "LIDAR" "memory" "v.in.ascii" "valgrind"; for emails from
> Radim, Helena, and myself.

None of those emails provide an answer if and when the problem may be
fixed. I can't find it at least. Maybe it is there, but I don't see it.

> > Second, if this is a problem that has come up repeatedly, this is an
> > indication that work is needed in this area.
> 
> I think we all agree that is true. But we don't know of a good solution
> yet.

Thanks,
Maciek

--------------------
W polskim Internecie s± setki milionów stron. My przekazujemy Tobie tylko najlepsze z nich!
http://katalog.epf.pl/