[fdo-users] Improving performance for NEN1878 provider, howto?
Piet Bibber
piet.bibber at gmail.com
Thu Oct 4 14:23:11 EDT 2007
Thanks Dan,
The BigPerformanceTest resulted in the following results:
(For each class I test for null and get the attribute)
BigPerformanceTests.select (10/04/07 20:18:52):
880 features from class Symbolen in 0.047 seconds
4302 features from class Bebouwing in 0.219 seconds
390 features from class TopoLijnen in 0.016 seconds
3276 features from class InfraLand in 0.171 seconds
620 features from class Terreinen in 0.032 seconds
712 features from class InfraWater in 0.031 seconds
1867 features from class StraatHuis in 0.094 seconds
22 features from class OverigeNamen in 0.000 seconds
4.375 seconds
This looks really good. The 4.375 seconds include fetching the schema (the
time spend here is no problem)
I will try to get other UnitTest's to run as well.
// Piet
On 10/4/07, Dan Stoica <dan.stoica at autodesk.com> wrote:
>
> *"What else can I do to trace performance or to make GetData faster?"*
>
>
>
> Do you have a profiling tool to run against a simple FDO test? If not, use
> some timers and narrow down the bottleneck. E.g.
>
> - How long takes SpatialExtents(), Count()
>
> - How long takes IsNull()
>
> - How long takes GetGeometry()
>
>
>
> Dan.
>
>
>
>
>
> *From:* fdo-users-bounces at lists.osgeo.org [mailto:
> fdo-users-bounces at lists.osgeo.org] *On Behalf Of *Badreddine Karoui
> *Sent:* Thursday, October 04, 2007 9:20 AM
> *To:* FDO Users Mail List
> *Subject:* RE: [fdo-users] Improving performance for NEN1878 provider,
> howto?
>
>
>
> Try replace the SHP provider dll in the providers xml file with you
> provider's dll and see if that makes a difference. Keep the provider name as
> is. Now, Map will think your provider is the SHP provider and you will need
> to provide the connection parameters through the SHP connect dialog or just
> hard code them in your connection object.
>
>
>
> Badreddine
>
>
>
> *From:* fdo-users-bounces at lists.osgeo.org [mailto:
> fdo-users-bounces at lists.osgeo.org] *On Behalf Of *Piet Bibber
> *Sent:* Thursday, October 04, 2007 6:17 AM
> *To:* FDO Users Mail List
> *Subject:* Re: [fdo-users] Improving performance for NEN1878 provider,
> howto?
>
>
>
> Dan,
>
>
>
> SelectAggregates does instanciate the OptimizedAggregateReader (at the
> bottom of the Execute command).
>
>
>
> Where I believe it is very slow is in getting data (through the IsNull and
> GetData methods);
>
> The dialog with "Polulating Feature Cache" is on screen during 3 to 5
> seconds; the same dataset in sdf takes sub seconds to load. The " Polulating
> Feature Cache" dialog doesn't even appear.
>
> My features are all in memory, no disk access is needed, yet is multiple
> times slower.
>
>
>
> i must be doing something wrong.
>
>
>
> When I return true to IsNull (and avoid the GetData call), its still
> slow....
>
>
>
> What else can I do to trace performance or to make GetData faster?
>
>
>
> Thanks for the help so far
>
> Piet
>
>
>
>
>
>
> On 10/1/07, *Dan Stoica* <dan.stoica at autodesk.com> wrote:
>
> Indeed, OptimizedAggregateReader is the place. Make sure your
> SelectAggregates command is instantiating such a reader.
>
>
>
> Dan.
>
>
>
> *From:* fdo-users-bounces at lists.osgeo.org [mailto:
> fdo-users-bounces at lists.osgeo.org] *On Behalf Of *Piet Bibber
> *Sent:* Monday, October 01, 2007 10:16 AM
> *To:* FDO Users Mail List
> *Subject:* Re: [fdo-users] Improving performance for NEN1878 provider,
> howto?
>
>
>
> Robert, Badreddine, Gavin
>
>
>
> I do provide the extends and counts in Nen1878OptimizedAggregateReader. Is
> that what you refer to?
>
> Piet
>
>
>
>
>
> On 10/1/07, *Robert Fortin* <robert.fortin at autodesk.com> wrote:
>
> Badreddine,
>
> Isn't it the combination of count and spatialExtents that need to be
> optimized?
> I believe Map looks for the equivalent of
>
> Select SpatialExtents(geometry), count(*) from <class>
>
> RF
>
>
> -----Original Message-----
> From: fdo-users-bounces at lists.osgeo.org
> [mailto:fdo-users-bounces at lists.osgeo.org] On Behalf Of Badreddine
> Karoui
> Sent: Monday, October 01, 2007 9:42 AM
> To: fdo-users at lists.osgeo.org
> Subject: RE: [fdo-users] Improving performance for NEN1878 provider,
> howto?
>
> Hi,
>
> You may want to implement the FdoISelectAggregates command plus the
> SpatialExtents and Count functions. If these two functions are not
> implemented, then Map will scan all the geometries to find the minimum
> bounding rectangle of all the data.
>
> Badreddine
>
> -----Original Message-----
> From: fdo-users-bounces at lists.osgeo.org
> [mailto:fdo-users-bounces at lists.osgeo.org] On Behalf Of Piet Bibber
> Sent: Monday, October 01, 2007 7:46 AM
> To: fdo-users at lists.osgeo.org
> Subject: [fdo-users] Improving performance for NEN1878 provider, howto?
>
>
> Hello,
>
> My NEN1878 provider is almost done, but performance is not so fantastic
> when
> reading the geometry; reading the schema is alright.
> That is stange, as I cache in the geometry when I read the schema
> (NEN1878
> schema discovery requires a full file scan; no seperate schema header).
> (The
> schema is read within 3 seconds for a 2.3MB file, including the geometry
> and
> attributes (all in memory (using STL)).
> The SDF and SHP all seem so fast :-)
>
> Map 3D seem to spend most of his time caching in the geometry (see
> source
> code below)
>
> I based my provider on the SHP provider, the reader seem to use
> FDO_READER
> template, but I can't find where that points to. Maybe there are better
> optimized readers.
>
> Thanks.
> Piet
>
> Excerpt from the source files:
>
> FdoByteArray * CreateCachedGeometry(FdoByteArray* fgfByteArray_I)
> {
> FdoByteArray* fgfByteArray = fgfByteArray_I; // it may be
> reallocated
> int count = mFeature->GetNumPoints();
> FdoInt32 numPointOrds = 2;
> FdoInt32 numBytesForOrdinates = numPointOrds *
> sizeof(double)
> * count;
> FdoInt32 numParts = 0;
> FdoGeometryType fdoGeomType = GetFdoGeometryType();
> FdoInt32 dimensionality = FdoDimensionality_XY;
> FdoInt32 numPositions = mFeature->GetNumPoints();
>
> fgfByteArray->Clear();
>
> fgfByteArray = FdoByteArray::Append(fgfByteArray,
> sizeof(fdoGeomType), (FdoByte *) &fdoGeomType);
>
> switch (fdoGeomType)
> {
> case FdoGeometryType_None:
> break;
>
> case FdoGeometryType_Point:
> fgfByteArray = FdoByteArray::Append(fgfByteArray,
> sizeof(dimensionality), (FdoByte*)&dimensionality);
> fgfByteArray = FdoByteArray::Append(fgfByteArray,
> numBytesForOrdinates, (FdoByte*)mFeature->GetPoints());
> break;
>
> case FdoGeometryType_LineString:
> fgfByteArray = FdoByteArray::Append(fgfByteArray,
> sizeof(dimensionality), (FdoByte *) &dimensionality);
> fgfByteArray = FdoByteArray::Append(fgfByteArray,
> sizeof(count),
> (FdoByte *)&count);
> fgfByteArray = FdoByteArray::Append(fgfByteArray,
> numBytesForOrdinates, (FdoByte *)mFeature->GetPoints());
> break;
>
> default:
> throw
> FdoException::Create(FdoException::NLSGetMessage(FDO_NLSID(FDO_10_UNSUPP
> ORTEDGEOMETRYTYPE)));
> }
>
> return (fgfByteArray);
> }
>
>
> // Returns pointer to vector of DoublePoints
> inline DoublePoint* Nen1878Feature::GetPoints()
> {
> return (&mCoordinates[0]);
> }
>
> --
> View this message in context:
> http://www.nabble.com/Improving-performance-for-NEN1878-provider%2C-howt
> o--tf4547817s18162.html#a12977669
> Sent from the fdo-users mailing list archive at Nabble.com<http://nabble.com/>
> .
>
> _______________________________________________
> fdo-users mailing list
> fdo-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-users
>
> _______________________________________________
> fdo-users mailing list
> fdo-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-users
>
> _______________________________________________
> fdo-users mailing list
> fdo-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-users
>
>
>
>
> _______________________________________________
> fdo-users mailing list
> fdo-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-users
>
>
>
> _______________________________________________
> fdo-users mailing list
> fdo-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/fdo-users/attachments/20071004/fab9a8f0/attachment.html
More information about the fdo-users
mailing list