[GRASS-dev] [GRASS GIS] #1694: r.in.lidar tries to allocate way too much memory

[GRASS GIS]

#1694: r.in.lidar tries to allocate way too much memory
------------------------+---------------------------------------------------
 Reporter:  torsti      |       Owner:  grass-dev@…              
     Type:  defect      |      Status:  new                      
 Priority:  normal      |   Milestone:  7.0.0                    
Component:  Raster      |     Version:  svn-trunk                
 Keywords:  r.in.lidar  |    Platform:  Linux                    
      Cpu:  x86-64      |  
------------------------+---------------------------------------------------

Comment(by torsti):

 So memory allocation is based on the extent of the region and not the
 bounding box of the LAS data, that explains a lot. That was my mistake
 there! Still, to be on the safe side I've included the more detailed
 information that was asked for.

 g.region -p
 {{{
 D2/2: G__read_Cell_head
 D2/2: G__read_Cell_head_array
 D2/2: G__read_Cell_head
 D2/2: G__read_Cell_head_array
 projection: 1 (UTM)
 zone:       35
 datum:      etrs89
 ellipsoid:  grs80
 north:      7776450.217
 south:      6605838.902
 west:       61686.152
 east:       732907.723
 nsres:      1.00000027
 ewres:      0.99999936
 rows:       1170611
 cols:       671222
 cells:      785739856642
 }}}

 lasinfo R4133C4.laz
 {{{
 ---------------------------------------------------------
   Header Summary
 ---------------------------------------------------------

   Version:                     1.2
   Source ID:                   0
   Reserved:                    0
   Project ID/GUID:             '00000000-0000-0000-0000-000000000000'
   System ID:                   ''
   Generating Software:         'EspaEngine'
   File Creation Day/Year:      0/0
   Header Byte Size             227
   Data Offset:                 329
   Header Padding:              2
   Number Var. Length Records:  1
   Point Data Format:           1
   Number of Point Records:     11064863
   Compressed:                  True
   Compression Info:            LASzip Version 2.1r0 c2 50000: POINT10 2
 GPSTIME11 2
   Number of Points by Return:  0 0 0 0 0
   Scale Factor X Y Z:          0.01 0.01 0.01
   Offset X Y Z:                -0.00 -0.00 -0.00
   Min X Y Z:                   389000.00 7149000.00 91.36
   Max X Y Z:                   391999.99 7151999.99 139.61
   Spatial Reference:
 None

 None

 ...

 }}}

 I updated r.in.lidar to revision
 [https://trac.osgeo.org/grass/changeset/52593 52593].

 both r.in.lidar and  r.in.xyz complain about the amount of memory, because
 the region is too big, but the amount of memory they ask for is not in the
 exabyte range.

 {{{
 > r.in.lidar -o --overwrite input=R4133C4.laz output=R4133C4.las
 method=mean

 D2/2: G__read_Cell_head
 D2/2: G__read_Cell_head_array
 Over-riding projection check
 D2/2: region.n=7776450.217000  region.s=6605838.902000
 region.ns_res=1.000000
 D2/2: region.rows=1170611  [box_rows=1170611]  region.cols=671222
 Current region rows: 1170611, cols: 671222
 ERROR: G_calloc: unable to allocate 785741027253 * 4 bytes of memory at
        main.c:534
 }}}

 {{{
 > las2txt --keep-classes 2 --parse xyz --delimiter="|" --input R4133C4.las
 --output=/tmp/las.tmp
 > r.in.xyz input=/tmp/las.tmp output=R4133C4_ground_points

 D2/2: G__read_Cell_head
 D2/2: G__read_Cell_head_array
 D2/2: region.n=7776450.217000  region.s=6605838.902000
 region.ns_res=1.000000
 D2/2: region.rows=1170611  [box_rows=1170611]  region.cols=671222
 Current region rows: 1170611, cols: 671222
 ERROR: G_calloc: unable to allocate 785741027253 * 4 bytes of memory at
        main.c:491
 }}}

 After adjusting the extent to the BBOX of the LAS data:
 {{{
 g.region -a n=7152000 s=7149000 e=392000 w=389000 res=1
 }}}

 r.in.lidar
 {{{
 D2/2: G__read_Cell_head
 D2/2: G__read_Cell_head_array
 Over-riding projection check
 D2/2: region.n=7152000.000000  region.s=7149000.000000
 region.ns_res=1.000000
 D2/2: region.rows=3000  [box_rows=3000]  region.cols=3000
 Reading data ...
 D2/2: pass=1/1  pass_n=7152000.000000  pass_s=7149000.000000  rows=3000
 D2/2: allocating n_array
 D2/2: allocating sum_array
  100%
 D2/2: pass 1 finished, 11064827 coordinates in box
 Writing to map ...
  100%
 D1/2: close R4133C4.las compressed
 D1/2: G_find_raster2(): name=R4133C4.las mapset=PERMANENT
 D1/2: G_find_raster2(): name=R4133C4.las mapset=PERMANENT
 D1/2: G_find_raster2(): name=R4133C4.las mapset=PERMANENT
 r.in.lidar complete. 11064827 points found in region.
 D1/2: Processed 11064863 points
 }}}

 r.in.xyz
 {{{
 D2/2: G__read_Cell_head
 D2/2: G__read_Cell_head_array
 D2/2: region.n=7152000.000000  region.s=7149000.000000
 region.ns_res=1.000000
 D2/2: region.rows=3000  [box_rows=3000]  region.cols=3000
 D2/2: estimated number of lines in file: 3224929
 Reading data ...
 D2/2: pass=1/1  pass_n=7152000.000000  pass_s=7149000.000000  rows=3000
 D2/2: allocating n_array
 D2/2: allocating sum_array
  100%
 D2/2: pass 1 finished, 3135351 coordinates in box
 Writing to map ...
  100%
 D1/2: close R4133C4_ground_points compressed
 D1/2: G_find_raster2(): name=R4133C4_ground_points mapset=PERMANENT
 D1/2: G_find_raster2(): name=R4133C4_ground_points mapset=PERMANENT
 D1/2: G_find_raster2(): name=R4133C4_ground_points mapset=PERMANENT
 r.in.xyz complete. 3135351 points found in region.
 D1/2: Processed 3135367 lines.
 }}}

 So everything seems to work now.

 Sorry for the most likely unfounded & inaccurate bug report.

 Cheers,
 Torsti

-- 
Ticket URL: <https://trac.osgeo.org/grass/ticket/1694#comment:8>
GRASS GIS <http://grass.osgeo.org>