[mapguide-users] Unmanaged SHP File Locations

Mon May 28 16:38:52 EDT 2007

Zak,
Yes, I understand now.  If that is the case and FDO is churning through all
of those shape files each time any single class is opened, then that's where
the efficiency and performance problem most likely lies.

Andy 

-----Original Message-----
From: mapguide-users-bounces at lists.osgeo.org
[mailto:mapguide-users-bounces at lists.osgeo.org] On Behalf Of Zak James
Sent: Monday, May 28, 2007 1:17 PM
To: MapGuide Users Mail List
Subject: Re: [mapguide-users] Unmanaged SHP File Locations

Andy,

I could be completely wrong about this, but I think that when you call
Open() on the feature source, it creates an XML configuration for all the
data in the feature source on the fly. In that case, it wouldn't matter that
your layer only references a single class, there is some overhead to having
all those shapefiles. It also looks like having shapefiles in different
projections within one feature source could make things worse as it has to
reproject them to figure out the overall extents.

I'm going to do some experiments to see if I can see the same problems
- you've definitely got me interested... and maybe someone who knows more
about FDO will make some comment and educate all of us :)

zak

On 5/25/07, Andy Morsell <amorsell at spatialgis.com> wrote:
> Zak,
> In this case, I loaded the exact some files into the repository 
> (managed) that I had in the unmanaged directory to see the differences in
performance.
> So the performance difference must be attributed to something internal 
> to MapGuide / FDO.
>
> Pointing at a directory of shapefiles is done at the unmanaged data 
> source alias configuration level in webconfig.ini.  When you create a 
> layer, that layer still only references one feature class (one shape 
> file) from within that directory so there should be no reason for the 
> provider to do extra work when accessing layers for a map render since 
> it knows what exact shape file to access.  So there's something else 
> going on here........ Sure, when you are first defining a layer, it 
> must enumerate all shape files in a directory so you can choose which 
> feature source to use, but that's a one time only thing.
>
>
> Andy
>
> -----Original Message-----
> From: mapguide-users-bounces at lists.osgeo.org
> [mailto:mapguide-users-bounces at lists.osgeo.org] On Behalf Of Zak James
> Sent: Friday, May 25, 2007 5:39 PM
> To: MapGuide Users Mail List
> Subject: Re: [mapguide-users] Unmanaged SHP File Locations
>
> There may be efficiencies to be gained in the provider, but there are 
> some limitations that don't have easy solutions.
>
> There are no attribute indexes for shapefiles so a lookup in a large 
> file is going to be slow.
>
> Pointing at a directory of shapefiles is like pointing at one big data 
> source with many feature classes. It may be convenient during 
> development, but for production you will want to reference specific 
> shapefiles or at least break down the large directory into smaller 
> sets. I'm pretty sure managed shapefile feature sources are one per
directory.
>
> If an unmanaged shape feature source draws more slowly than a managed 
> one, it's worth checking that the shx file is present and valid.
>
> zak
>
> On 5/25/07, Andy Morsell <amorsell at spatialgis.com> wrote:
> >
> >
> > First, I love the unmanaged resource concept and will most likely 
> > deploy most future MGOS applications this way.
> >
> > But, similar to Chris' problems, I have discovered that displaying 
> > tooltips from unmanaged shape files causes a much more significant 
> > performance hit and delay.  In my case, I have a 150 MB parcel 
> > polygon shape file for a county.  In the unmanaged case, each 
> > tooltip AJAX fetch spikes the CPU with mgserver.exe at 100% for a 
> > couple of seconds.  Doing the same with the exact same shape file in 
> > a managed state (not converted to SDF at load time), the CPU will 
> > only hit 25% or
> less and responds much more quickly.
> >
> > Further, I discovered that the more unmanaged data in a map, the 
> > slower the tooltip response is for all layers.  For instance, if I 
> > add another 7 or so layers (like streets, water bodies, etc) to the 
> > map, when I float over a parcel polygon it can take up to 10 seconds 
> > for a tooltip response during which time mgserver is cranking away.  
> > So, I'm not sure what it's doing at that point.  All of these 
> > additional layers are all in the same unmanaged data source folder.
> >
> > I went ahead and loaded all of the shape files into the repository 
> > and created a duplicate map pointing to the new managed feature sources.
> > This map behaves fine with tooltip fetches consuming 25% of a single 
> > CPU and very quick response time.  So, this seems to confirm that 
> > the more unmanaged feature sources in a map (at least from the same 
> > directory), the larger the performance hit and the slower the map 
> > responds
> to requests.
> >
> > I have appended this information to the TRAC ticket that Chris 
> > started earlier today.
> >
> >
> > Andy Morsell, P.E.
> > Spatial Integrators, Inc.
> > http://www.SpatialGIS.com
> >  ________________________________
> >  From: mapguide-users-bounces at lists.osgeo.org
> > [mailto:mapguide-users-bounces at lists.osgeo.org] On Behalf Of Chris 
> > Gountanis
> > Sent: Friday, May 25, 2007 7:52 AM
> > To: 'MapGuide Users Mail List'
> > Subject: [mapguide-users] Unmanaged SHP File Locations
> > Importance: High
> >
> >
> >
> >
> >
> >
> > Why do unmanaged FeatureSources respond slower as more files get 
> > added to the directory location? We have a FeatureSource called 
> > SHPDefualt for example. The zooms take longer as well as other 
> > functions. It seems when you query a Feature Source is does not hit 
> > the DBF you want directly it seems to fiddle with everything in the 
> > unmanaged directory. We love the concept of a folder that contains 
> > all SHP files. One location is great for batch processes and great 
> > for editing new layers. It is just simple for customers to 
> > understand. For sure easier to back up and recreate. Will this always be
a performance killer?
> >
> >
> >
> >
> >
> > Is this a bug? Will this be fixed soon?
> >
> >
> >
> >
> >
> > This really takes the complication out of knowing what the heck is 
> > going on in the repository. Here is XML Example:
> >
> > <?xml version="1.0" encoding="utf-8" ?>
> >
> > <FeatureSource
> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > xmlns:xsd="http://www.w3.org/2001/XMLSchema"
> > xsi:noNamespaceSchemaLocation="FeatureSource-1.0.0.xsd">
> >
> > <Provider>OSGeo.SHP</Provider>
> >
> > <Parameter>
> >
> > <Name>DefaultFileLocation</Name>
> >
> > <Value>C:\SHPFILES</Value>
> >
> > </Parameter>
> >
> > </FeatureSource>
> >
> >
> >
> >
> >
> >
> >
> >
> > Chris
> >
> >
> > _______________________________________________
> > mapguide-users mailing list
> > mapguide-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapguide-users
> >
> >
>
>
> --
> Zak James
> Applications and Software Development
> DM Solutions Group Inc.
> http://www.dmsolutions.ca
> _______________________________________________
> mapguide-users mailing list
> mapguide-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-users
>
>
> _______________________________________________
> mapguide-users mailing list
> mapguide-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-users
>

--
Zak James
Applications and Software Development
DM Solutions Group Inc.
http://www.dmsolutions.ca
_______________________________________________
mapguide-users mailing list
mapguide-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapguide-users