[mapguide-internals] Release Build not working???? Debug build is fine!!! Architectural Solution: RFC 63

Wed Apr 22 08:08:08 EDT 2009

UV,

What are the specs of your server?  What are you using for the various memory settings in serverconfig.ini?

Martin

-----Original Message-----
From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of UV
Sent: Wednesday, April 22, 2009 12:30 AM
To: MapGuide Internals Mail List
Subject: Re: [mapguide-internals] Release Build not working???? Debug build is fine!!! Architectural Solution: RFC 63

clearly our EAP Map does not fit into memory.... so there is disk access 
for the rendering and for the tiles.

Thanks for the advice about for the number of worker threads...   which 
server classes are dealing with this?
(I wanted to ask Bruce to augment his very instructive architectural 
description with real classnames at some point.
Without a reference to the actual code base its not easy to find things 
there and it remains a more abstract description than necessary.)

I am now looking at a number of configuration parameters. I gonna try to 
adjust the threadpoolsize.
Probably the data connection cache can be tuned also.
This is for sure a useful way to make a deployment work for a specific 
map on a specific server.

However, this RFC is about providing additional parameters to tune the 
server based on a better understanding of its inner workings.
It would increase the efficiency of the resource use in many scenarios 
in a more flexible way than the simple throtteling down.
This is not an urgent request anyway... just brainstorming for now.

Firstly we have to find out why out of memory exceptions dont fail the 
tile generation.....
Avoiding the memory exhaustion seems not the right solution for this 
behaviour.

Thanks for the comments.
UV

Traian Stanev wrote:
> Is it thrashing when saving all the tile images or when rendering maps? In the former case there really isn't much you can do -- even if you make giant tiles with MapGuide, you still have to split them up into small tiles and save them to disk. I guess the way to test that would be to render all the tiles without saving them to disk and see if it is still thrashing.
>
> Traian
>
> ________________________________________
> From: mapguide-internals-bounces at lists.osgeo.org [mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Zac Spitzer [zac.spitzer at gmail.com]
> Sent: Tuesday, April 21, 2009 10:44 PM
> To: MapGuide Internals Mail List
> Subject: Re: [mapguide-internals] Release Build not working???? Debug build is  fine!!! Architectural Solution: RFC 63
>
> In my experience mapguide always thrashes the disk when tiling using my seeder
> which is always requesting the same scale and area
>
> depending on the way the data is read, unless you have a very small
> dataset, the I/O cache
> is going to be probably too small...
>
> watching the IO reads / writes under task manager I often see
> 100-200mbs (ish approx )
> a second on a raid 10 array..
>
> in 32-bit land there isn't going to be a lot of memory for the I/O cache
>
> z
>
> On Wed, Apr 22, 2009 at 12:30 PM, Traian Stanev
> <traian.stanev at autodesk.com> wrote:
>   
>> I don't see how the analogy with Windows copying is right. If you have 10 threads reading the same piece of data from disk, it's most likely that the data is in the I/O cache, so no disk reads are really done after the first one. The other 9 threads would get the data in memory for free.
>>
>> Have you tried reducing the number of simultaneous threads in your server thread pool in order to alleviate your low memory condition?
>>
>>
>> Traian
>>
>> ________________________________________
>> From: mapguide-internals-bounces at lists.osgeo.org [mapguide-internals-bounces at lists.osgeo.org] On Behalf Of UV [uvwild at gmail.com]
>> Sent: Tuesday, April 21, 2009 10:17 PM
>> To: MapGuide Internals Mail List
>> Subject: Re: [mapguide-internals] Release Build not working???? Debug build     is fine!!! Architectural Solution: RFC 63
>>
>> I finally came to the conclusion that this has been caused by resource
>> limits. (the  infamous Out Of Memory exception)
>>
>> A memory limitation causes the mapping / rendering service to abort
>> rendering the tile before all features are determined.
>> Its very interesting that the serialization enforced by STDOUT of the
>> debug build solves this problem already!
>>
>> Firstly this is a bug because the faulty Tile gets returned without an
>> error message. (like #462)
>>
>> However, this gives us as reason to suggest a redesign of the Tiling
>> service.
>> Basically, the server does not do anything efficient if we start to
>> request mutliple tiles of the same map and scale at the same time.
>> This is like windows copying. If you start n parallel copy streams
>> between the same disks you end up thrashing your disk.
>>
>> In our mapguide case this means that the server should maintain a
>> request queue and serialize request to the same map and scale
>> and maybe even compute 4 - 16 tiles at once and slice them up on the
>> bitmap level.
>> This does not seem very hard to do and will improve performance and
>> stability of the mapguide server dramatically.
>>
>> See also RFC 63
>>
>> UV wrote:
>>     
>>> Hi,
>>>
>>> now for the second time I tried to run my 8GB map sample in a release
>>> build.
>>> In the debug build the server renders all tiles just fine.
>>>
>>> When I did the same test with the release build the server created
>>> only tiles in a single color - no more features....
>>> (the sea is blue.... the desert is beige) the good news is that its
>>> deterministic..... the same wrong tiles each time.
>>>
>>> I ran the test suite.... OK 430 tests
>>> The server INFO Trace.log does not show anything special.
>>>
>>> The server is executed from the exact same location in the default
>>> installation directory.... I verified the DLL versions....
>>> No error messages... the access.log shows the error free access to the
>>> tiles..... but the computed tiles are crap.
>>>
>>> Anyone seen this before?
>>> Any idea what to look for? As its a release build and the debug
>>> version is working sorting this out is a challenging task.
>>> If there were any error messages this would be useful but there aint any.
>>>
>>> PS:
>>> The server repeatedly creates an additional directory in the tilecache
>>> called C-0 (as opposed to C0)
>>> containing also very boring featureless tiles  named 0_-1.png to
>>> 9_-1.png. This looks like a bug.
>>> _______________________________________________
>>> mapguide-internals mailing list
>>> mapguide-internals at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>>
>>>       
>> _______________________________________________
>> mapguide-internals mailing list
>> mapguide-internals at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>> _______________________________________________
>> mapguide-internals mailing list
>> mapguide-internals at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>
>>     
>
>
>
> --
> Zac Spitzer -
> http://zacster.blogspot.com
> +61 405 847 168
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>   

_______________________________________________
mapguide-internals mailing list
mapguide-internals at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapguide-internals