[mapserver-users] logging GDAL queries
Sebastian E. Ovide
sebastian.ovide at gmail.com
Sat Aug 14 04:49:11 PDT 2010
Hi Ivan,
the pyramid levels in Oracle are not exactly the same as the zoom levels in
MapServer... Each level whould have 1/4 number of pixels as the previous
level...ie: X/2*Y/2
The size of the pyramid image at each level is determined by the original
image size and the pyramid level, according to the following formulas:
r(n) = (int)(r(0) / 2^n)
c(n) = (int)(c(0) / 2^n)
In my case I have generated all the possible levels of pyramids... The Idea
of generating the pyramids is to speeding up the zoomed out images.. ie: if
you want to display the whole UK, you do not need to get the 100kx160k
pixels... you can read an upper level in the pyramid reading only 1kx1k
pixels... in the same way, if you need to do a zoom in at street level, you
can read from the level 0, that means reading the maximum resolution and it
is fast as you read only a fraction of the whole image...
with MapViewer+GDAL it getting very slow when generating tiles at street
level .... it should not be.... at that point you do not need pyramid any
more...
If I didn't have created pyramids, it still would be fast at street level
but it would be very slow at city level and it would be impossible to
display at country level...
does it make sense ?
On Fri, Aug 13, 2010 at 6:57 PM, Ivan Lucena <ivan.lucena at pmldnet.com>wrote:
> Sebastian,
>
> You said:
>
> > yep I did... and the weir thing is that generating the images from "far
> > away" zoom under 12... it performs very well... not sure why it doesn't
> > in the levels 13-17.... for that reason I was trying to find the SQL
> > queries....
>
> But your GeoRaster Pyramid level is <maxLevel>11</maxLevel>
>
> By the size of you image you should have 20 level.
>
> Can you run sdo_geor.generatePyramid() again changing only the number of
> levels, keeping the same method. That should run faster, keeping the
> existing pyramid levels.
>
> Please let me know how it works now.
>
> Regards,
>
> Ivan
>
> > -------Original Message-------
> > From: Ivan Lucena <ivan.lucena at pmldnet.com>
> > To: Sebastian E. Ovide <sebastian.ovide at gmail.com>
> > Cc: mapserver-users at lists.osgeo.org
> > Subject: Re: [mapserver-users] logging GDAL queries
> > Sent: Aug 13 '10 11:16
> >
> > Sebastian,
> >
> > > -------Original Message-------
> > > From: Sebastian E. Ovide <sebastian.ovide at gmail.com>
> > > To: Ivan Lucena <ivan.lucena at pmldnet.com>
> > > Cc: mapserver-users at lists.osgeo.org
> > > Subject: Re: [mapserver-users] logging GDAL queries
> > > Sent: Aug 13 '10 09:27
> > >
> > > On Fri, Aug 13, 2010 at 1:59 PM, Ivan Lucena <[LINK:
> > > mailto:ivan.lucena at pmldnet.com] ivan.lucena at pmldnet.com> wrote:
> > > Sebastian,
> > >
> > > You are using a very small block size, maybe because you are also
> using
> > > ArcGIS products and they only support Oracle GeoRaster with 256x256
> > > blocking. Is that right?
> > >
> > >
> > > nop... I guess that 256x256 if the default gdal_translate block size
> ?
> >
> > That is the default on GDAL/GeoRaster driver but you can change it using
> -co BLOCKXSIZE and -co BLOCKYSIZE.
> >
> > But that would require you to load the image again (and generate
> Pyramids again...)
> >
> > >
> > > Because of that, to access your GeoRaster object, the GDAL driver is
> > > requesting a very large array of BLOB (1000000 approximately). That
> can
> > > make your sever such-up memory from the machine where it is running
> and it
> > > is probably using virtual memory (disk) what is really slow.
> > >
> > > do you mean where MapServer is running ?
> > >
> > > top - 14:41:31 up 7 days, 6:20, 10 users, load average: 2.81, 1.92,
> > > 1.20
> > > Tasks: 340 total, 2 running, 338 sleeping, 0 stopped, 0 zombie
> > > Cpu(s): 11.8%us, 10.4%sy, 7.7%ni, 62.1%id, 0.0%wa, 0.0%hi,
> > > 8.0%si, 0.0%st
> > > Mem: 12331240k total, 5992136k used, 6339104k free, 189416k
> > > buffers
> > > Swap: 9084716k total, 703708k used, 8381008k free, 1481992k
> > > cached
> > >
> > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> > > COMMAND
> > > 24850 www-data 20 0 533m 330m 18m R 14 2.7 0:00.94
> > > mapserv
> > > 24852 www-data 20 0 532m 329m 18m S 19 2.7 0:01.22
> > > mapserv
> > > 24849 www-data 20 0 529m 326m 18m S 16 2.7 0:01.00
> > > mapserv
> > > 24851 www-data 20 0 517m 314m 18m S 13 2.6 0:00.96
> > > mapserv
> > > 24853 www-data 20 0 499m 296m 18m S 8 2.5 0:00.75
> > > mapserv
> > > 24854 www-data 20 0 470m 267m 18m S 5 2.2 0:00.53
> > > mapserv
> >
> > Yeah. I am sorry. I called Mapserver a client :)
> >
> > It is a Oracle client in that case. You probably have another nice
> machine running Oracle.
> >
> > That is not the problem.
> >
> > >
> > > It also uses some memory on the client side just to hold the array
> too
> > > but if you are running everything (Mapserver,Oracle) on the same
> machine
> > > that will adds up.
> > >
> > >
> > > Oracle is running on another Machine... and is very likely that it is
> the
> > > bottle neck...
> >
> > Maybe not but I believe you have tools on Enterprise Manager to show the
> volume and speed of traffic.
> >
> > >
> > >
> > > We are aware of that problem but it would be very helpful if you
> could
> > > file a bug report on GDAL's trac [[LINK: http://trac.osgeo.org/gdal/
> ]
> > > http://trac.osgeo.org/gdal/] Thanks.
> > >
> > >
> > > I guess that the problem is that MapServer triys to read the whole
> map
> > > instead of reading only tile ???? is that correct ?
> > > Just as curiosity. Did you loaded that image with gdal_translate?
> How
> > > long it took and long does it takes to generate pyramids? Ha, you
> forgot to
> > > say what version of Oracle you are using.
> > >
> > >
> > > I loaded it with gdal in around 5 minutes... the pyramid (all levels)
> has
> > > been created overnight....
> >
> > You original file was 51 Gb and it was loaded in 5 minutes !
> >
> > See, forget all about network traffic. That is just great.
> >
> > Note that it doesn't matter if you generate pyramids on PL/SQL or
> gdaladdo, the GDAL driver will call the same PL/SQL statement that you would
> write yourself.
> >
> > So I believe that the problem is that Mapserver need to call GDALOpen()
> several times on that GeoRaster and that makes the GDAL driver to run the
> main-big-query several times. That needs to be addressed some how.
> >
> > Regards,
> >
> > Ivan
> >
> >
> >
> >
> >
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> >
>
--
Sebastian E. Ovide
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20100814/cb249f7d/attachment.htm>
More information about the MapServer-users
mailing list