[mapguide-internals] Generating Xml for "Select Features"

[Trevor Wekel]

Hi Kenneth,

I was thinking about this a bit and you may be able to implement a
pseudo stream by deriving from MgByteSourceImpl.

Read() -> would perform a ReadNext operation and do some internal
buffering
GetLength() -> return zero
IsRewindable() -> return false
Rewind() -> throw MgInvalidOperation exception

Then you can create the reader using MgByteReader(MgByteSource*
byteSource).  I think ProxyFeatureReader::ToXml() currently returns a
rewindable reader with a known length.  If we switch to more of a true
streaming representation, some of the code calling ToXml() could break.
That's something to investigate.

Thanks,
Trevor 

-----Original Message-----
From: mapguide-internals-bounces at lists.osgeo.org
[mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of
Kenneth, GEOGRAF A/S
Sent: Wednesday, June 27, 2007 2:11 AM
To: MapGuide Internals Mail List
Subject: Re: [mapguide-internals] Generating Xml for "Select Features"

Ok, I agree that raising the limit would be a bad idea.
The "write to file" solution is better, but still requires the server to

buffer the entire resultset.
This will both require the entire resultset to be in memory at some 
time, and also delay the initial partial response.

I guess a single row should fit in 16Mb, so perhaps it would be enough 
to override the ProxyFeatureReader::ToXml(),
so it can also take a stream. Would it break the API to have an 
overloaded version, but not expose it?

I would use the stream method within the HttpHandler only, leaving it up

to someone else to bring it into the other languages.
For the regular ToXml(), a file backed serialization would probably 
improve performance.
Can you give me a pointer to some code that uses temporary files, so 
that I may implement it in a standardized fashion?

Regards, Kenneth, GEOGRAF A/S



Trevor Wekel skrev:
> Hi Kenneth,
>
> ToXml() is marked as EXTERNAL_API in MgReader so changing the
signature
> could break some code.  Also, all PUBLISHED_API and EXTERNAL_API
within
> MapGuide have to function correctly for all 3 web languages - PHP,
.Net,
> and Java.  Implementing API support for native streaming in all three
> languages would be a non-trivial effort.
>
> To get around the 16MB limit, write the string to a file and use new
> MgByteSource("filename", true).  I would only do this is the string is
> too large.
>
> Increasing the 16MB limit on MgByte is probably not a good idea,
> especially for this API.  Selecting an entire feature source (like the
> US roads network) into an in-memory XML blob would be bad.
>
> Thanks,
> Trevor
>
>
> -----Original Message-----
> From: mapguide-internals-bounces at lists.osgeo.org
> [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of
> Kenneth, GEOGRAF A/S
> Sent: Tuesday, June 26, 2007 5:26 AM
> To: MapGuide Internals Mail List
> Subject: [mapguide-internals] Generating Xml for "Select Features"
>
> When using the "Select Features" from the Http API, the result comes 
> back in Xml.
> The call chain is:
>
> ProxyFeatureReader::ToXml()
> MgByteSource::MgByteSource(char* data, int len)
> ByteSourceMemoryImpl::ByteSourceMemoryImpl(char* data, int len)
> MgByte::MgByte(char* data, int len, AllocatorType allocator)
>
> When the size of the Xml to return grows, MgByte throws an exception
in 
> line 34:
>     if (len > MgByte::MaxSize && allocator != MgByte::None)
>     {
>         throw new MgArgumentOutOfRangeException(L"MgByte.MgByte", 
> __LINE__, __WFILE__, NULL, L"", NULL);
>     }
>
> This happens because the ByteSourceMemoryImpl does not pass an 
> AllocatorType argument, and thus defaults to MgByte::None.
> A simple solution would be to fix this error in ByteSourceMemoryImpl, 
> but I don't have enough overview to see if this has other
implications.
>
> I will happily submit a patch for ByteSourceMemoryImpl, but is that an

> acceptable solution?
>
> I would prefer if the code was changed, so that 
> ProxyFeatureReader::ToXml() took a stream as input, and wrote to that 
> stream.
> That would also improve response/perfomance time, as the current 
> implementation buffers the entire result, and also uses the
std::string 
> + operator, which has a high overhead for large strings.
>
> Is the second solution something that could be accepted as a patch? I 
> so, I will happily submit a patch for it.
>
>   
_______________________________________________
mapguide-internals mailing list
mapguide-internals at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapguide-internals