[fdo-internals] SHP performance problem may be due to .idx

[BTenBroeck]

It seems that the provider does rewrite the idx files on access. And in my
particular case, it not only rewrites them, but at times rewrites them with
0 bytes, and then prevents access to the map at large because a single layer
cannot be rendered. 

I tried the read-only approach suggested by Dave, but if I make the files
(either idx, or the entire set) ro, then Mapguide pegs CPU usage for minutes
at a time, and again, the maps fail to render. 

I'm using an unmanaged datasource as the files are updated regularly by
other agencies.



Dave Wilson wrote
> This behavior has been a bug going back over 4+ years when I first
> reported it internally and I didn't get a good answer back then for the
> reason why it might be an issue. I have been (on Windows) marking my files
> as read only and the files are no longer modified on a read. This is not a
> new bug. This has been happening as far back as I can remember in the
> provider. 
> 
> Regards,
> Dave
> 
> -----Original Message-----
> From: 

> fdo-internals-bounces at .osgeo

>  [mailto:

> fdo-internals-bounces at .osgeo

> ] On Behalf Of Dan Stoica
> Sent: Tuesday, October 09, 2012 9:15 AM
> To: FDO Internals Mail List
> Subject: Re: [fdo-internals] SHP performance problem may be due to .idx
> 
> It is not a bug.
> 
> Greg is right, the idx file timestamp is changed but the file is not
> recreated. I didn't measure but I doubt that just updating the header can
> introduce such penalty.
> 
> The only thing I can think of is Ticket #776: SHP provider: spatial index
> out of date is not recreated, submitted in 2011.
> 
> 	In ShpFileSet::GetSpatialIndex() it reads:
> 
>             		// Check the timestamps for the SI and the SHP. The IDX file
> is always modified after SHP
>            		// but it can get stale when the SHP has been edited by 3th
> party applications.
> 
> Could you please check, before doing Validate, that the IDX file timestamp
> is as recent as the SHP file timestamp? I don't have MG to confirm your
> findings, I just ran a SHP provider unit test in debug.
> 
> I didn't understand the " writing a total of 900 idx files" part. There
> should be only 30 idx files. 
> 
> Regards.
> 
> 
> -----Original Message-----
> From: 

> fdo-internals-bounces at .osgeo

>  [mailto:

> fdo-internals-bounces at .osgeo

> ] On Behalf Of everling
> Sent: Saturday, October 06, 2012 4:50 AM
> To: 

> fdo-internals at .osgeo

> Subject: Re: [fdo-internals] SHP performance problem may be due to .idx
> 
> Is that a confirmation that it is a bug? I hope that this is an issue that
> may be fixed.
> 
> When I do a Validate on a SHP FeatureSource with 30 SHP files, MGOS 2.4
> RC2 will recreate all 30 idx files for each SHP file, writing a total of
> 900 idx files when it did not have to write anything at all. Even marking
> the idx files as read-only did not mitigate this undesirable behaviour.
> 
> I apologise if I may be whiny, but it is quite painful to realise that
> MGOS is rewriting the idx files needlessly and having to wait long minutes
> for MGOS to stop chasing its tail to get a validation report, or to do
> anything else with SHP files where MGOS is determined to rewrite the idx
> files.
> 
> On an additional note, I have checked a MapGuide Enterprise 2011
> installation to discover that it does not rewrite the idx files
> needlessly.
> Is this a possible hint of regression?
> 
> Thank you.
> 
> 
> 
> --
> View this message in context:
> http://osgeo-org.1560.n6.nabble.com/SHP-performance-problem-may-be-due-to-idx-tp5003321p5006817.html
> Sent from the FDO Internals mailing list archive at Nabble.com.
> _______________________________________________
> fdo-internals mailing list

> fdo-internals at .osgeo

> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> _______________________________________________
> fdo-internals mailing list

> fdo-internals at .osgeo

> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> _______________________________________________
> fdo-internals mailing list

> fdo-internals at .osgeo

> http://lists.osgeo.org/mailman/listinfo/fdo-internals





--
View this message in context: http://osgeo-org.1560.n6.nabble.com/SHP-performance-problem-may-be-due-to-idx-tp5003321p5043025.html
Sent from the FDO Internals mailing list archive at Nabble.com.