Re: [mapserver-users] logging GDAL queries

[Ivan Lucena]

Sebastian,

You said:

> yep I did... and the weir thing is that generating the images from "far
> away" zoom under 12... it performs very well... not sure why it doesn't
> in the levels 13-17....  for that reason I was trying to find the SQL
> queries.... 

But your GeoRaster Pyramid level is <maxLevel>11</maxLevel> 

By the size of you image you should have 20 level.

Can you run sdo_geor.generatePyramid() again changing only the number of levels, keeping the same method. That should run faster, keeping the existing pyramid levels. 

Please let me know how it works now.

Regards,

Ivan

>  -------Original Message-------
>  From: Ivan Lucena <ivan.lucena at pmldnet.com>
>  To: Sebastian E. Ovide <sebastian.ovide at gmail.com>
>  Cc: mapserver-users at lists.osgeo.org
>  Subject: Re: [mapserver-users] logging GDAL queries
>  Sent: Aug 13 '10 11:16
>  
>  Sebastian,
>  
>  >  -------Original Message-------
>  >  From: Sebastian E. Ovide <sebastian.ovide at gmail.com>
>  >  To: Ivan Lucena <ivan.lucena at pmldnet.com>
>  >  Cc: mapserver-users at lists.osgeo.org
>  >  Subject: Re: [mapserver-users] logging GDAL queries
>  >  Sent: Aug 13 '10 09:27
>  >  
>  >  On Fri, Aug 13, 2010 at 1:59 PM, Ivan Lucena <[LINK:
>  >  mailto:ivan.lucena at pmldnet.com] ivan.lucena at pmldnet.com> wrote:
>  >  Sebastian,
>  >  
>  >  You are using a very small block size, maybe because you are also using
>  >  ArcGIS products and they only support Oracle GeoRaster with 256x256
>  >  blocking. Is that right?
>  >  
>  >  
>  >  nop... I guess that 256x256 if the default gdal_translate block size ?
>  
>  That is the default on GDAL/GeoRaster driver but you can change it using -co BLOCKXSIZE and -co BLOCKYSIZE.
>  
>  But that would require you to load the image again (and generate Pyramids again...)
>  
>  >   
>  >  Because of that, to access your GeoRaster object, the GDAL driver is
>  >  requesting a very large array of BLOB (1000000 approximately). That can
>  >  make your sever such-up memory from the machine where it is running and it
>  >  is probably using virtual memory (disk) what is really slow.
>  >  
>  >  do you mean where MapServer is running ?
>  >  
>  >  top - 14:41:31 up 7 days,  6:20, 10 users,  load average: 2.81, 1.92,
>  >  1.20
>  >  Tasks: 340 total,   2 running, 338 sleeping,   0 stopped,   0 zombie
>  >  Cpu(s): 11.8%us, 10.4%sy,  7.7%ni, 62.1%id,  0.0%wa,  0.0%hi, 
>  >  8.0%si,  0.0%st
>  >  Mem:  12331240k total,  5992136k used,  6339104k free,   189416k
>  >  buffers
>  >  Swap:  9084716k total,   703708k used,  8381008k free,  1481992k
>  >  cached
>  >  
>  >    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ 
>  >  COMMAND
>  >  24850 www-data  20   0  533m 330m  18m R   14  2.7   0:00.94
>  >  mapserv
>  >  24852 www-data  20   0  532m 329m  18m S   19  2.7   0:01.22
>  >  mapserv
>  >  24849 www-data  20   0  529m 326m  18m S   16  2.7   0:01.00
>  >  mapserv
>  >  24851 www-data  20   0  517m 314m  18m S   13  2.6   0:00.96
>  >  mapserv
>  >  24853 www-data  20   0  499m 296m  18m S    8  2.5   0:00.75
>  >  mapserv
>  >  24854 www-data  20   0  470m 267m  18m S    5  2.2   0:00.53
>  >  mapserv
>  
>  Yeah. I am sorry. I called Mapserver a client :)
>  
>  It is a Oracle client in that case. You probably have another nice machine running Oracle.
>  
>  That is not the problem.
>  
>  >  
>  >     It also uses some memory on the client side just to hold the array too
>  >  but if you are running everything (Mapserver,Oracle) on the same machine
>  >  that will adds up.
>  >  
>  >  
>  >  Oracle is running on another Machine... and is very likely that it is the
>  >  bottle neck...
>  
>  Maybe not but I believe you have tools on Enterprise Manager to show the volume and speed of traffic.
>  
>  >  
>  >  
>  >  We are aware of that problem but it would be very helpful if you could
>  >  file a bug report on GDAL's trac [[LINK: http://trac.osgeo.org/gdal/]
>  >  http://trac.osgeo.org/gdal/] Thanks.
>  >  
>  >  
>  >  I guess that the problem is that MapServer triys to read the whole map
>  >  instead of reading only tile ???? is that correct ?
>  >      Just as curiosity. Did you loaded that image with gdal_translate? How
>  >  long it took and long does it takes to generate pyramids? Ha, you forgot to
>  >  say what version of Oracle you are using.
>  >  
>  >  
>  >  I loaded it with gdal in around 5 minutes... the pyramid (all levels) has
>  >  been created overnight....
>  
>  You original file was 51 Gb and it was loaded in 5 minutes !
>  
>  See, forget all about network traffic. That is just great.
>  
>  Note that it doesn't matter if you generate pyramids on PL/SQL or gdaladdo, the GDAL driver will call the same PL/SQL statement that you would write yourself.
>  
>  So I believe that the problem is that Mapserver need to call GDALOpen() several times on that GeoRaster and that makes the GDAL driver to run the main-big-query several times. That needs to be addressed some how.
>  
>  Regards,
>  
>  Ivan
>  
>  
>  
>  
>  
>  _______________________________________________
>  mapserver-users mailing list
>  mapserver-users at lists.osgeo.org
>  http://lists.osgeo.org/mailman/listinfo/mapserver-users
>