[GRASS-dev] [GRASS GIS] #1662: Caching bug in 3D raster library with large data

[GRASS GIS]

#1662: Caching bug in 3D raster library with large data
-------------------------------------+--------------------------------------
 Reporter:  huhabla                  |       Owner:  grass-dev@…              
     Type:  defect                   |      Status:  new                      
 Priority:  critical                 |   Milestone:  7.0.0                    
Component:  Raster3D                 |     Version:  svn-trunk                
 Keywords:  3D raster, tiles, cache  |    Platform:  Linux                    
      Cpu:  x86-32                   |  
-------------------------------------+--------------------------------------

Comment(by huhabla):

 Many thanks for the backtrace Anna. I was able to reproduce this behavior
 with my shiny "new" 32Bit Atom Netbook.
 The problem is that the cache file accessed in cache.c line 310 gets wrong
 so that a corrupted
 tile index is read from the file. This index is much to large for the tile
 array accessed in cachehash.c line 114,
 hence crash because of memory violation.

 The cache file is corrupted because it exceedes the 32Bit limit of 4GB,
 hence the computed offset in cache.c
 faces a number overrun, resulting in a wrong file offsets. The size of
 size_t and off_t is 4 Bytes on my 32 Bit system
 and therefore not suited for files larger than 4GB.

 IMHO its a problem of the 32Bit operating system. I don't know how to
 solve this issue ... LFS?? Any suggestions?

 Conclusion: raster3D maps with more than a billion cells in case of type
 float, or more than 500 million cells in case
 of type double are not supported on 32Bit systems.

 Replying to [comment:5 annakrat]:
 > Here is the backtrace:
 >
 > {{{
 > #0  0xb7f7c3ac in Rast3d_cache_hash_name2index (h=0x8051f98,
 name=1715470336) at cachehash.c:114
 > #1  0xb7f7adf7 in Rast3d_cache_elt_ptr (c=0x805a0f0, name=1715470336) at
 cache1.c:469
 > #2  0xb7f7b04c in Rast3d_cache_load (c=0x805a0f0, name=1715470336) at
 cache1.c:517
 > #3  0xb7f7c08b in Rast3d_flush_all_tiles (map=0x8054028) at cache.c:310
 > #4  0x0804bd57 in test_large_file_random (depths=20, rows=5400,
 cols=10800, tile_size=32) at test_put_get_value_large_file.c:283
 > #5  0x0804b421 in unit_test_put_get_value_large_file (depths=20,
 rows=5400, cols=10800, tile_size=32) at test_put_get_value_large_file.c:39
 > #6  0x08049c34 in main (argc=2, argv=0xbfffedd4) at test_main.c:160
 > }}}
 >
 > Thank you for the instructions, here they are for completeness:
 >
 > {{{
 > gdb test.g3d.lib
 >
 > r unit=large
 > ..... segfault
 > bt
 > }}}
 >

-- 
Ticket URL: <http://trac.osgeo.org/grass/ticket/1662#comment:6>
GRASS GIS <http://grass.osgeo.org>