[GRASS-dev] How many points in a point layer and v.surf.bspline questions

Tue Jan 4 10:42:09 EST 2011

>>>Large file support does not help here because 8.5 billion points
>>>exceeds the number of supported features in GRASS vectors which is
>>>about 2 billion (2,147,483,647 to be precise). 
        >>I'll chop the inputfile into sections less than 2 billion then. 
Is there any reason that the 2 billion limit on GRASS vectors >>cannot be 
raised? ( variable change vs types of variable and lots of ugly fixes in 
various places?) If it's something simple, I can play >>with it, but I'm 
not a C programmer :-(. 

>The limit can be raised, but, unfortunately, this is not simple, because 
portability across platforms must be maintained. It boils down to the 
problem that it is not simple to have a portable >64 bit integer type on 
at least Linux, Mac, Windows, both 32 bit and 64 bit architectures. I 
would like to have a portable 64 bit integer type for grass 7 though...

I see GRASS7 in my future :-)

In the short term, Can this limit be gotten around by putting the data in 
Postgresql/Postgis/Spatialite and using v.external or v.db.connect for the 
inputpoints for v.surf.bspline?

Doug

Doug Newcomb 
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newcomb at fws.gov
---------------------------------------------------------------------------------------------------------
The opinions I express are my own and are not representative of the 
official policy of the U.S.Fish and Wildlife Service or Dept. of the 
Interior.   Life is too short for undocumented, proprietary data formats.

Markus Metz <markus.metz.giswork at googlemail.com> 
01/04/2011 04:43 AM

To
Doug_Newcomb at fws.gov
cc
grass-dev at lists.osgeo.org, Markus Neteler <neteler at osgeo.org>
Subject
Re: [GRASS-dev] How many points in a point layer and v.surf.bspline 
questions

On Mon, Jan 3, 2011 at 4:01 PM, <Doug_Newcomb at fws.gov> wrote:

Markus M., 

>Large file support does not help here because 8.5 billion points
>exceeds the number of supported features in GRASS vectors which is
>about 2 billion (2,147,483,647 to be precise). 
        I'll chop the inputfile into sections less than 2 billion then. Is 
there any reason that the 2 billion limit on GRASS vectors cannot be 
raised? ( variable change vs types of variable and lots of ugly fixes in 
various places?) If it's something simple, I can play with it, but I'm not 
a C programmer :-(. 

The limit can be raised, but, unfortunately, this is not simple, because 
portability across platforms must be maintained. It boils down to the 
problem that it is not simple to have a portable 64 bit integer type on at 
least Linux, Mac, Windows, both 32 bit and 64 bit architectures. I would 
like to have a portable 64 bit integer type for grass 7 though...

Markus M

>Further on, a region with 6.8 billion cells is a bit large since the
>interpolated raster will be held in memory which would require about
>50 GB RAM (that could be fixed for v.surf.bspline by keeping 
>intermediate data on disk). 
 I know some folks with computers with 64GB+ of RAM, so that is not an 
insurmountable issue. However, it would probably be better to go the 
intermediate route.   

> No idea where the request for
>18446744071812941729 * 8 bytes comes from, this is astronomical. 
That was my reaction.  Perhaps the computer was screaming in pain.:-)

>Apart from that, spline steps of 40 seem ok, since the point distance
>is 5 m or less, spline steps of 20 would also be ok. The larger the
>spline steps, the smoother the resulting surface. Smoothing is also
>controlled through lambda.

Thanks for the insight, I will need to change my approach to minimize the 
smoothing. 

Doug 

Markus M 
Doug Newcomb             
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newcomb at fws.gov
---------------------------------------------------------------------------------------------------------
The opinions I express are my own and are not representative of the 
official policy of the U.S.Fish and Wildlife Service or Dept. of the 
Interior.   Life is too short for undocumented, proprietary data formats. 

Markus Metz <markus.metz.giswork at googlemail.com> 
01/03/2011 09:13 AM 

To
Markus Neteler <neteler at osgeo.org> 
cc
Doug_Newcomb at fws.gov, grass-dev at lists.osgeo.org 
Subject
Re: [GRASS-dev] How many points in a point layer and v.surf.bspline 
questions

On Mon, Jan 3, 2011 at 2:57 PM, Markus Neteler <neteler at osgeo.org> wrote:
> hi Doug,
>
> On Mon, Jan 3, 2011 at 2:26 PM,  <Doug_Newcomb at fws.gov> wrote:
>>
>> Hi Folks,
>>
>> I aggregated all of the bare earth lidar points for the state of North
>> Carolina into a single file  Imported all 8.5 billion points for NC 
into one
>> point layer with no topology.  I was sort of curious to see if I could
>> generate a seamless 20 ft elevation grid for the State of North 
Carolina
>> from a single data layerusing bspline and  tried the following command:
>>
>> GRASS 6.5.svn (ncstpft_nad83):~ > v.surf.bspline input=all_nc_be_pts2
>> raster=all_nc_be_20ft_bspline sie=40 sin=40 layer=0
>> WARNING: Coor files of vector map <all_nc_be_pts2 at statewide> is larger 
than
>>          it should be (158913789952 bytes excess)
>> Cells for raster map <all_nc_be_20ft_bspline> will be interpolated
>> ERROR: G_calloc: unable to allocate 18446744071812941729 * 8 bytes of
>>        memory at dalloc.c:66
>
> you are hitting the limits :) I dunno if large file support helps, but
> for vector data
> it is only available in GRASS 7 to my knowledge.
>
Large file support does not help here because 8.5 billion points
exceeds the number of supported features in GRASS vectors which is
about 2 billion (2,147,483,647 to be precise).

Further on, a region with 6.8 billion cells is a bit large since the
interpolated raster will be held in memory which would require about
50 GB RAM (that could be fixed for v.surf.bspline by keeping
intermediate data on disk). No idea where the request for
18446744071812941729 * 8 bytes comes from, this is astronomical.

Apart from that, spline steps of 40 seem ok, since the point distance
is 5 m or less, spline steps of 20 would also be ok. The larger the
spline steps, the smoother the resulting surface. Smoothing is also
controlled through lambda.

Markus M

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/grass-dev/attachments/20110104/6251aadc/attachment-0001.html