[GRASS-dev] Re: [GRASS GIS] #1625: Disk performance degrades by several orders of magnitude on two processes

[GRASS GIS]

#1625: Disk performance degrades by several orders of magnitude on two processes
---------------------+------------------------------------------------------
 Reporter:  sprice   |       Owner:  grass-dev@…              
     Type:  defect   |      Status:  new                      
 Priority:  normal   |   Milestone:  7.0.0                    
Component:  Default  |     Version:  svn-trunk                
 Keywords:           |    Platform:  MacOSX                   
      Cpu:  x86-64   |  
---------------------+------------------------------------------------------

Comment(by huhabla):

 To assure that this issue is related to GRASS and not to your
 implementation you to need to check the IO performance of a native GRASS
 module running in parallel.

 I have attached (multiproc_mapcalc.py) a simple grass module to check the
 IO disk performance by running multiple instances of r.mapcalc in
 parallel. The implementation assures disk syncing at the end of the
 processing and measures the time between start and end of processing and
 sync in seconds. I did not face large performance drops running multiple
 r.mapcalc instances in parallel. Here some examples running on my AMD 6
 core 1TB HD Ubuntu 64Bit system:

 Running a single r.mapcalc instance to create a 50.000.000 cell integer
 raster map:
 {{{
 GRASS 7.0.svn (TestLL):~/ > python multiproc_mapcalc.py base=raster
 nprocs=1 size=50
 projection: 3 (Latitude-Longitude)
 zone:       0
 datum:      wgs84
 ellipsoid:  wgs84
 north:      80N
 south:      0
 west:       0
 east:       62:30E
 nsres:      0:00:36
 ewres:      0:00:36
 rows:       8000
 cols:       6250
 cells:      50000000
 ### main process ###
 process id: 4152
 ### sub process for map <raster_0> ###
 ppid 4152 pid 4158
  100%
 Sync disk
 Time for processing: 3.885329 seconds
 Removing raster <raster_0>
 }}}

 Running six r.mapcalc instance to create six 50.000.000 cell integer
 raster maps:
 {{{
 GRASS 7.0.svn (TestLL):~/ > python multiproc_mapcalc.py base=raster
 nprocs=6 size=50
 projection: 3 (Latitude-Longitude)
 zone:       0
 datum:      wgs84
 ellipsoid:  wgs84
 north:      80N
 south:      0
 west:       0
 east:       62:30E
 nsres:      0:00:36
 ewres:      0:00:36
 rows:       8000
 cols:       6250
 cells:      50000000
 ### main process ###
 process id: 4175
 ### sub process for map <raster_1> ###
 ppid 4175 pid 4182
 ### sub process for map <raster_0> ###
 ppid 4175 pid 4181
 ### sub process for map <raster_2> ###
 ppid 4175 pid 4183
 ### sub process for map <raster_3> ###
 ppid 4175 pid 4186
 ### sub process for map <raster_4> ###
 ppid 4175 pid 4187
 ### sub process for map <raster_5> ###
 ppid 4175 pid 4190
  100%
  100%
  100%
  100%
  100%
  100%
 Sync disk
 Time for processing: 4.878238 seconds
 Removing raster <raster_0>
 Removing raster <raster_1>
 Removing raster <raster_2>
 Removing raster <raster_3>
 Removing raster <raster_4>
 Removing raster <raster_5>
 }}}

-- 
Ticket URL: <https://trac.osgeo.org/grass/ticket/1625#comment:3>
GRASS GIS <http://grass.osgeo.org>