<br><tt><font size=2>>>>Large file support does not help here
because 8.5 billion points<br>
>>>exceeds the number of supported features in GRASS vectors which
is<br>
>>>about 2 billion (2,147,483,647 to be precise).</font></tt><font size=3>
</font>
<br><tt><font size=2> >>I'll chop the
inputfile into sections less than 2 billion then. Is there any reason that
the 2 billion limit on GRASS vectors >>cannot be raised? ( variable
change vs types of variable and lots of ugly fixes in various places?)
If it's something simple, I can play >>with it, but I'm not a C programmer
:-(. </font></tt>
<br>
<br><font size=3>>The limit can be raised, but, unfortunately, this
is not simple, because portability across platforms must be maintained.
It boils down to the problem that it is not simple to have a portable >64
bit integer type on at least Linux, Mac, Windows, both 32 bit and 64 bit
architectures. I would like to have a portable 64 bit integer type for
grass 7 though...</font>
<br>
<br><font size=3>I see GRASS7 in my future :-)</font>
<br>
<br><font size=3>In the short term, Can this limit be gotten around by
putting the data in Postgresql/Postgis/Spatialite and using v.external
or v.db.connect for the inputpoints for v.surf.bspline?</font>
<br>
<br><font size=3>Doug</font>
<br>
<br>
<br>
<br><font size=2 face="sans-serif">Doug Newcomb
<br>
USFWS<br>
Raleigh, NC<br>
919-856-4520 ext. 14 doug_newcomb@fws.gov<br>
---------------------------------------------------------------------------------------------------------<br>
The opinions I express are my own and are not representative of the official
policy of the U.S.Fish and Wildlife Service or Dept. of the Interior.
Life is too short for undocumented, proprietary data formats.</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>Markus Metz <markus.metz.giswork@googlemail.com></b>
</font>
<p><font size=1 face="sans-serif">01/04/2011 04:43 AM</font>
<td width=59%>
<table width=100%>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td><font size=1 face="sans-serif">Doug_Newcomb@fws.gov</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td><font size=1 face="sans-serif">grass-dev@lists.osgeo.org, Markus Neteler
<neteler@osgeo.org></font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">Re: [GRASS-dev] How many points in a
point layer and v.surf.bspline questions</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br>
<br><font size=3>On Mon, Jan 3, 2011 at 4:01 PM, <</font><a href=mailto:Doug_Newcomb@fws.gov><font size=3 color=blue><u>Doug_Newcomb@fws.gov</u></font></a><font size=3>>
wrote:</font>
<br><font size=2 face="sans-serif"><br>
Markus M.,</font><font size=3> </font>
<br><tt><font size=2><br>
>Large file support does not help here because 8.5 billion points<br>
>exceeds the number of supported features in GRASS vectors which is<br>
>about 2 billion (2,147,483,647 to be precise).</font></tt><font size=3>
</font>
<br><tt><font size=2> I'll chop the inputfile
into sections less than 2 billion then. Is there any reason that the 2
billion limit on GRASS vectors cannot be raised? ( variable change vs types
of variable and lots of ugly fixes in various places?) If it's something
simple, I can play with it, but I'm not a C programmer :-(. </font></tt>
<br>
<br><font size=3>The limit can be raised, but, unfortunately, this is not
simple, because portability across platforms must be maintained. It boils
down to the problem that it is not simple to have a portable 64 bit integer
type on at least Linux, Mac, Windows, both 32 bit and 64 bit architectures.
I would like to have a portable 64 bit integer type for grass 7 though...<br>
<br>
Markus M<br>
</font>
<br><tt><font size=2><br>
>Further on, a region with 6.8 billion cells is a bit large since the<br>
>interpolated raster will be held in memory which would require about<br>
>50 GB RAM (that could be fixed for v.surf.bspline by keeping</font></tt><font size=3>
</font><tt><font size=2><br>
>intermediate data on disk).</font></tt><font size=3> </font>
<br><tt><font size=2> I know some folks with computers with 64GB+
of RAM, so that is not an insurmountable issue. However, it would probably
be better to go the intermediate route. </font></tt><font size=3>
</font>
<br><tt><font size=2><br>
> No idea where the request for<br>
>18446744071812941729 * 8 bytes comes from, this is astronomical.</font></tt><font size=3>
</font>
<br><tt><font size=2>That was my reaction. Perhaps the computer was
screaming in pain.:-)</font></tt>
<br><tt><font size=2><br>
<br>
>Apart from that, spline steps of 40 seem ok, since the point distance<br>
>is 5 m or less, spline steps of 20 would also be ok. The larger the<br>
>spline steps, the smoother the resulting surface. Smoothing is also<br>
>controlled through lambda.</font></tt>
<br><tt><font size=2><br>
Thanks for the insight, I will need to change my approach to minimize the
smoothing.</font></tt><font size=3> <br>
</font><tt><font size=2><br>
Doug</font></tt><font size=3> <br>
<br>
<br>
</font><tt><font size=2><br>
<br>
Markus M</font></tt><font size=3> </font>
<br><font size=2 face="sans-serif">Doug Newcomb
<br>
USFWS<br>
Raleigh, NC<br>
919-856-4520 ext. 14 </font><a href=mailto:doug_newcomb@fws.gov target=_blank><font size=2 color=blue face="sans-serif"><u>doug_newcomb@fws.gov</u></font></a><font size=2 face="sans-serif"><br>
---------------------------------------------------------------------------------------------------------<br>
The opinions I express are my own and are not representative of the official
policy of the U.S.Fish and Wildlife Service or Dept. of the Interior.
Life is too short for undocumented, proprietary data formats.</font><font size=3>
<br>
<br>
</font>
<table width=100%>
<tr valign=top>
<td width=42%><font size=1 face="sans-serif"><b>Markus Metz <</b></font><a href=mailto:markus.metz.giswork@googlemail.com target=_blank><font size=1 color=blue face="sans-serif"><b><u>markus.metz.giswork@googlemail.com</u></b></font></a><font size=1 face="sans-serif"><b>></b>
</font>
<p><font size=1 face="sans-serif">01/03/2011 09:13 AM</font><font size=3>
</font>
<td width=57%>
<br>
<table width=100%>
<tr valign=top>
<td width=9%>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td width=90%><font size=1 face="sans-serif">Markus Neteler <</font><a href=mailto:neteler@osgeo.org target=_blank><font size=1 color=blue face="sans-serif"><u>neteler@osgeo.org</u></font></a><font size=1 face="sans-serif">></font><font size=3>
</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td><a href=mailto:Doug_Newcomb@fws.gov target=_blank><font size=1 color=blue face="sans-serif"><u>Doug_Newcomb@fws.gov</u></font></a><font size=1 face="sans-serif">,
</font><a href="mailto:grass-dev@lists.osgeo.org" target=_blank><font size=1 color=blue face="sans-serif"><u>grass-dev@lists.osgeo.org</u></font></a><font size=3>
</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">Re: [GRASS-dev] How many points in a
point layer and v.surf.bspline questions</font></table>
<br>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br><font size=3><br>
<br>
</font><tt><font size=2><br>
On Mon, Jan 3, 2011 at 2:57 PM, Markus Neteler <</font></tt><a href=mailto:neteler@osgeo.org target=_blank><tt><font size=2 color=blue><u>neteler@osgeo.org</u></font></tt></a><tt><font size=2>>
wrote:<br>
> hi Doug,<br>
><br>
> On Mon, Jan 3, 2011 at 2:26 PM, <</font></tt><a href=mailto:Doug_Newcomb@fws.gov target=_blank><tt><font size=2 color=blue><u>Doug_Newcomb@fws.gov</u></font></tt></a><tt><font size=2>>
wrote:<br>
>><br>
>> Hi Folks,<br>
>><br>
>> I aggregated all of the bare earth lidar points for the state
of North<br>
>> Carolina into a single file Imported all 8.5 billion points
for NC into one<br>
>> point layer with no topology. I was sort of curious to see
if I could<br>
>> generate a seamless 20 ft elevation grid for the State of North
Carolina<br>
>> from a single data layerusing bspline and tried the following
command:<br>
>><br>
>> GRASS 6.5.svn (ncstpft_nad83):~ > v.surf.bspline input=all_nc_be_pts2<br>
>> raster=all_nc_be_20ft_bspline sie=40 sin=40 layer=0<br>
>> WARNING: Coor files of vector map <all_nc_be_pts2@statewide>
is larger than<br>
>> it should be
(158913789952 bytes excess)<br>
>> Cells for raster map <all_nc_be_20ft_bspline> will be interpolated<br>
>> ERROR: G_calloc: unable to allocate 18446744071812941729 * 8 bytes
of<br>
>> memory at dalloc.c:66<br>
><br>
> you are hitting the limits :) I dunno if large file support helps,
but<br>
> for vector data<br>
> it is only available in GRASS 7 to my knowledge.<br>
><br>
Large file support does not help here because 8.5 billion points<br>
exceeds the number of supported features in GRASS vectors which is<br>
about 2 billion (2,147,483,647 to be precise).<br>
<br>
Further on, a region with 6.8 billion cells is a bit large since the<br>
interpolated raster will be held in memory which would require about<br>
50 GB RAM (that could be fixed for v.surf.bspline by keeping<br>
intermediate data on disk). No idea where the request for<br>
18446744071812941729 * 8 bytes comes from, this is astronomical.<br>
<br>
Apart from that, spline steps of 40 seem ok, since the point distance<br>
is 5 m or less, spline steps of 20 would also be ok. The larger the<br>
spline steps, the smoother the resulting surface. Smoothing is also<br>
controlled through lambda.<br>
<br>
Markus M</font></tt><font size=3><br>
</font>
<br>
<br>