Re: [mapserver-users] logging GDAL queries

[Ivan Lucena]

Sebastian,

>  -------Original Message-------
>  From: Sebastian E. Ovide <sebastian.ovide at gmail.com>
>  To: Ivan Lucena <ivan.lucena at pmldnet.com>
>  Cc: mapserver-users at lists.osgeo.org
>  Subject: Re: [mapserver-users] logging GDAL queries
>  Sent: Aug 13 '10 09:27
>  
>  On Fri, Aug 13, 2010 at 1:59 PM, Ivan Lucena <[LINK:
>  mailto:ivan.lucena at pmldnet.com] ivan.lucena at pmldnet.com> wrote:
>  Sebastian,
>  
>  You are using a very small block size, maybe because you are also using
>  ArcGIS products and they only support Oracle GeoRaster with 256x256
>  blocking. Is that right?
>  
>  
>  nop... I guess that 256x256 if the default gdal_translate block size ?

That is the default on GDAL/GeoRaster driver but you can change it using -co BLOCKXSIZE and -co BLOCKYSIZE.

But that would require you to load the image again (and generate Pyramids again...)

>   
>  Because of that, to access your GeoRaster object, the GDAL driver is
>  requesting a very large array of BLOB (1000000 approximately). That can
>  make your sever such-up memory from the machine where it is running and it
>  is probably using virtual memory (disk) what is really slow.
>  
>  do you mean where MapServer is running ?
>  
>  top - 14:41:31 up 7 days,  6:20, 10 users,  load average: 2.81, 1.92,
>  1.20
>  Tasks: 340 total,   2 running, 338 sleeping,   0 stopped,   0 zombie
>  Cpu(s): 11.8%us, 10.4%sy,  7.7%ni, 62.1%id,  0.0%wa,  0.0%hi, 
>  8.0%si,  0.0%st
>  Mem:  12331240k total,  5992136k used,  6339104k free,   189416k
>  buffers
>  Swap:  9084716k total,   703708k used,  8381008k free,  1481992k
>  cached
>  
>    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ 
>  COMMAND
>  24850 www-data  20   0  533m 330m  18m R   14  2.7   0:00.94
>  mapserv
>  24852 www-data  20   0  532m 329m  18m S   19  2.7   0:01.22
>  mapserv
>  24849 www-data  20   0  529m 326m  18m S   16  2.7   0:01.00
>  mapserv
>  24851 www-data  20   0  517m 314m  18m S   13  2.6   0:00.96
>  mapserv
>  24853 www-data  20   0  499m 296m  18m S    8  2.5   0:00.75
>  mapserv
>  24854 www-data  20   0  470m 267m  18m S    5  2.2   0:00.53
>  mapserv

Yeah. I am sorry. I called Mapserver a client :)

It is a Oracle client in that case. You probably have another nice machine running Oracle.

That is not the problem.

>  
>     It also uses some memory on the client side just to hold the array too
>  but if you are running everything (Mapserver,Oracle) on the same machine
>  that will adds up.
>  
>  
>  Oracle is running on another Machine... and is very likely that it is the
>  bottle neck...

Maybe not but I believe you have tools on Enterprise Manager to show the volume and speed of traffic.

>  
>  
>  We are aware of that problem but it would be very helpful if you could
>  file a bug report on GDAL's trac [[LINK: http://trac.osgeo.org/gdal/]
>  http://trac.osgeo.org/gdal/] Thanks.
>  
>  
>  I guess that the problem is that MapServer triys to read the whole map
>  instead of reading only tile ???? is that correct ?
>      Just as curiosity. Did you loaded that image with gdal_translate? How
>  long it took and long does it takes to generate pyramids? Ha, you forgot to
>  say what version of Oracle you are using.
>  
>  
>  I loaded it with gdal in around 5 minutes... the pyramid (all levels) has
>  been created overnight....

You original file was 51 Gb and it was loaded in 5 minutes !
 
See, forget all about network traffic. That is just great.

Note that it doesn't matter if you generate pyramids on PL/SQL or gdaladdo, the GDAL driver will call the same PL/SQL statement that you would write yourself.

So I believe that the problem is that Mapserver need to call GDALOpen() several times on that GeoRaster and that makes the GDAL driver to run the main-big-query several times. That needs to be addressed some how.

Regards,

Ivan