[GRASS-user] v.clean process killed itselt!?

Nikos Alexandris nikos.alexandris at felis.uni-freiburg.de
Fri Jan 9 06:53:36 EST 2009 

Hi Markus.
Below more details upon the subject.

On Fri, 2009-01-09 at 09:36 +0100, Markus Neteler wrote:
> On Fri, Jan 9, 2009 at 6:27 AM, Nikos Alexandris
> <nikos.alexandris at felis.uni-freiburg.de> wrote:
> > Attempting to clean a very big vector map (after patching and before
> > dissolving) ended without success. It's about all CORINE tiles to form
> > the European-wide land cover map.
> >
> > The process was running for 2 days now and killed itself for reasons I
> > don't understand. The machine I currently work with has 4GB of RAM and
> > 8GB of swap memory, not to mention the free hard disk space.
> >
> > # the map is big!
> 
> I see two potential reasons:
> - large file support needed (--enable-largefile, did you use it?)
> - looking at the number of vertices, perhaps the process runs
>   out of memory (check 'dmesg' if you are on Linux). In this case
>   adding more swap space may suffice.

- Large file support is there. With the configuration I use to compile
the source code [1] LFS is reports "Yes" [2].

- So, it is a good practice to expand the swap space then (even more
than 8GB)? How much? Any method to practically estimate the potentially
needed space based on a big map's size?

  I believe that the relevant message confirms your skepsis Markus. See
[3]

[...]

> > -Additionally, I was watching from time to time the process's status via
> > "top".

> how much RAM/SWAP was used?

I have to admit that I was watching the %MEM column and not the VIRT
(which is I suppose the swap space). The %MEM, as long as I was keeping
an eye on it was between 25% and 39%, and at some point it quickly
reached 82% - 83%.

Anyhow, is it the only way to go about it? I repeat that all I want is
really to have a clean vector before dissolving.

> Markus

Kind regards, Nikos

=============================================================================
[1] CFLAGS="-g -Wall" LDFLAGS="-s" ./configure \
     --enable-64bit \
     --with-libs=/usr/lib64 \
     --with-cxx \
     --with-freetype=yes \
     --with-freetype-includes="/usr/include/freetype2/" \
     --with-postgres=no \
     --with-sqlite=yes \
     --enable-largefile=yes \
     --with-tcltk-includes="/usr/include/tcl8.4/" \
     --with-freetype-includes=/usr/include/freetype2 \
     --with-opengl-libs=/usr/include/GL \
     --with-readline \
     --with-python=yes \
     --with-proj-share=/usr/local/share/proj/ \
     --with-wxwidgets \
     --with-cairo \
     --with-ffmpeg=yes --with-ffmpeg-includes="/usr/include/ffmpeg/"
-----------------------------------------------------------------------------
[2] GRASS is now configured for:  x86_64-unknown-linux-gnu

  Source directory:            /usr/local/src/grass6_devel
  Build directory:             /usr/local/src/grass6_devel
  Installation directory:      ${prefix}/grass-6.4.svn
  Startup script in directory: ${exec_prefix}/bin
  C compiler:                  gcc -g -Wall 
  C++ compiler:                c++ -g -O2
  Building shared libraries:   yes
  64bit support:               yes
  OpenGL platform:             X11
  MacOSX application:         no

  NVIZ:                       yes

  BLAS support:               no
  C++ support:                yes
  Cairo support:              yes
  DWG support:                no
  FFMPEG support:             yes
  FFTW support:               yes
  FreeType support:           yes
  GDAL support:               yes
  GLw support:                no
  JPEG support:               yes
  LAPACK support:             no
  Large File support (LFS):   yes
  Motif support:              no
  MySQL support:              no
  NLS support:                no
  ODBC support:               no
  OGR support:                yes
  OpenGL support:             yes
  PNG support:                yes
  PostgreSQL support:         no
  Python support:             yes
  Readline support:           yes
  SQLite support:             yes
  Tcl/Tk support:             yes
  wxWidgets support:          yes
  TIFF support:               yes
  X11 support:                yes
-------------------------------------------------------------------------
[3] Taken from /var/log/messages

Jan  8 23:47:46 vertical -- MARK --
Jan  9 00:02:40 vertical kernel: [179692.069123] type=1503
audit(1231455736.104:5): operation="capable" name="sys_admin" pid=6911
profile="/usr/sbin/cupsd"
Jan  9 00:02:40 vertical kernel: [179692.069135] type=1503
audit(1231455736.104:6): operation="capable" name="sys_resource"
pid=6911 profile="/usr/sbin/cupsd"
Jan  9 00:02:40 vertical kernel: [179692.069141] type=1503
audit(1231455736.104:7): operation="capable" name="sys_rawio" pid=6911
profile="/usr/sbin/cupsd"
Jan  9 00:02:40 vertical kernel: [179692.069294] main-menu invoked
oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Jan  9 00:02:40 vertical kernel: [179692.069299] Pid: 6911, comm:
main-menu Tainted: P          2.6.27-9-generic #1
Jan  9 00:02:40 vertical kernel: [179692.069301] 
Jan  9 00:02:40 vertical kernel: [179692.069302] Call Trace:
Jan  9 00:02:40 vertical kernel: [179692.069312]  [<ffffffff802af87a>]
oom_kill_process+0x9a/0x230
Jan  9 00:02:40 vertical kernel: [179692.069316]  [<ffffffff802afd5f>] ?
select_bad_process+0xef/0x130
Jan  9 00:02:40 vertical kernel: [179692.069319]  [<ffffffff802aff35>]
out_of_memory+0x195/0x270
Jan  9 00:02:40 vertical kernel: [179692.069323]  [<ffffffff802b2d09>]
__alloc_pages_internal+0x4d9/0x520
Jan  9 00:02:40 vertical kernel: [179692.069328]  [<ffffffff802d58cd>]
alloc_pages_current+0xad/0x110
Jan  9 00:02:40 vertical kernel: [179692.069334]  [<ffffffff802ac617>]
__page_cache_alloc+0x67/0x80
Jan  9 00:02:40 vertical kernel: [179692.069338]  [<ffffffff802b646c>]
__do_page_cache_readahead+0xec/0x220
Jan  9 00:02:40 vertical kernel: [179692.069341]  [<ffffffff802ac710>] ?
sync_page+0x0/0x70
Jan  9 00:02:40 vertical kernel: [179692.069346]  [<ffffffff80267090>] ?
wake_bit_function+0x0/0x50
Jan  9 00:02:40 vertical kernel: [179692.069349]  [<ffffffff802b6603>]
do_page_cache_readahead+0x63/0x90
Jan  9 00:02:40 vertical kernel: [179692.069353]  [<ffffffff802add4a>]
filemap_fault+0x34a/0x430
Jan  9 00:02:40 vertical kernel: [179692.069357]  [<ffffffff802c2174>]
__do_fault+0x64/0x440
Jan  9 00:02:40 vertical kernel: [179692.069360]  [<ffffffff802c310e>]
handle_mm_fault+0x1ee/0x470
Jan  9 00:02:40 vertical kernel: [179692.069366]  [<ffffffff805053cf>]
do_page_fault+0x34f/0x750
Jan  9 00:02:40 vertical kernel: [179692.069370]  [<ffffffff802cda0e>] ?
free_pages_and_swap_cache+0x8e/0xb0
Jan  9 00:02:40 vertical kernel: [179692.069374]  [<ffffffff80234059>] ?
__phys_addr+0x9/0x50
Jan  9 00:02:40 vertical kernel: [179692.069380]  [<ffffffff803a7c78>] ?
__up_write+0x68/0x140
Jan  9 00:02:40 vertical kernel: [179692.069384]  [<ffffffff80502a7a>]
error_exit+0x0/0x70
Jan  9 00:02:40 vertical kernel: [179692.069386] 
Jan  9 00:02:40 vertical kernel: [179692.069387] Mem-Info:
Jan  9 00:02:40 vertical kernel: [179692.069389] Node 0 DMA per-cpu:
Jan  9 00:02:40 vertical kernel: [179692.069392] CPU    0: hi:    0,
btch:   1 usd:   0
Jan  9 00:02:40 vertical kernel: [179692.069394] CPU    1: hi:    0,
btch:   1 usd:   0
Jan  9 00:02:40 vertical kernel: [179692.069396] Node 0 DMA32 per-cpu:
Jan  9 00:02:40 vertical kernel: [179692.069399] CPU    0: hi:  186,
btch:  31 usd: 174
Jan  9 00:02:40 vertical kernel: [179692.069401] CPU    1: hi:  186,
btch:  31 usd: 185
Jan  9 00:02:40 vertical kernel: [179692.069403] Node 0 Normal per-cpu:
Jan  9 00:02:40 vertical kernel: [179692.069405] CPU    0: hi:  186,
btch:  31 usd: 172
Jan  9 00:02:40 vertical kernel: [179692.069407] CPU    1: hi:  186,
btch:  31 usd: 175
Jan  9 00:02:40 vertical kernel: [179692.069411] Active:583277
inactive:376079 dirty:0 writeback:0 unstable:0
Jan  9 00:02:40 vertical kernel: [179692.069412]  free:5383 slab:17065
mapped:182 pagetables:10765 bounce:0
Jan  9 00:02:40 vertical kernel: [179692.069415] Node 0 DMA free:9576kB
min:16kB low:20kB high:24kB active:0kB inactive:0kB present:8448kB
pages_scanned:0 all_unreclaimable? yes
Jan  9 00:02:40 vertical kernel: [179692.069419] lowmem_reserve[]: 0
2968 3976 3976
Jan  9 00:02:40 vertical kernel: [179692.069424] Node 0 DMA32
free:9980kB min:6016kB low:7520kB high:9024kB active:1868404kB
inactive:1040896kB present:3039764kB pages_scanned:6881497
all_unreclaimable? yes
Jan  9 00:02:40 vertical kernel: [179692.069429] lowmem_reserve[]: 0 0
1008 1008
Jan  9 00:02:40 vertical kernel: [179692.069433] Node 0 Normal
free:1976kB min:2040kB low:2548kB high:3060kB active:464704kB
inactive:463532kB present:1032192kB pages_scanned:2199846
all_unreclaimable? yes
Jan  9 00:02:40 vertical kernel: [179692.069438] lowmem_reserve[]: 0 0 0
0
Jan  9 00:02:40 vertical kernel: [179692.069442] Node 0 DMA: 2*4kB 4*8kB
6*16kB 3*32kB 4*64kB 3*128kB 4*256kB 1*512kB 3*1024kB 2*2048kB 0*4096kB
= 9576kB
Jan  9 00:02:40 vertical kernel: [179692.069453] Node 0 DMA32: 993*4kB
1*8kB 1*16kB 3*32kB 4*64kB 2*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB
1*4096kB = 9980kB
Jan  9 00:02:40 vertical kernel: [179692.069464] Node 0 Normal: 34*4kB
2*8kB 4*16kB 1*32kB 1*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB
0*4096kB = 1976kB
Jan  9 00:02:40 vertical kernel: [179692.069475] 1245 total pagecache
pages
Jan  9 00:02:40 vertical kernel: [179692.069476] 0 pages in swap cache
Jan  9 00:02:40 vertical kernel: [179692.069479] Swap cache stats: add
3126186, delete 3126186, find 729339/846430
Jan  9 00:02:40 vertical kernel: [179692.069481] Free swap  = 0kB
Jan  9 00:02:40 vertical kernel: [179692.069483] Total swap = 7813468kB
Jan  9 00:02:40 vertical kernel: [179692.088001] 1048576 pages RAM
Jan  9 00:02:40 vertical kernel: [179692.088003] 40728 pages reserved
Jan  9 00:02:40 vertical kernel: [179692.088005] 837 pages shared
Jan  9 00:02:40 vertical kernel: [179692.088006] 1000697 pages
non-shared
Jan  9 00:27:46 vertical -- MARK --

More information about the grass-user mailing list