[Gdal-dev] Re: Rendering optimization [Re: Looking for huge vector dataset]

[Mateusz Loskot]

Ari Jolma wrote:
> Mateusz Loskot kirjoitti:
>>> Has anybody seen rendering big vector data sets been
>>> satisfactorily solved in any desktop GIS?
>> It is a complex task, indeed. I've seen two approaches and the very
>> base idea is not directly related to rendering, but to data
>> management:
>> 
>> 1. load the whole geometry data into memory, into simple structures
>>  (arrays) - this is similar to how Manifold works
> 
> This is doomed to lead into problems once you try to open a theme
> with 5 million polygons...

Yup, in many cases it is.
However, both approaches introduce some problems.
In all-in-memory approach, the obvious assumption is a user
has a *lot* of RAM space, 4-8 GB.
But in turn, it provides the fastest access to features (accessing RAM
is faster than disk or network).

So, I'd conclude that everything depends on a particular application
and design (requirements).

>> 2. suck data from the disk (datasource like file/database) on every
>> read (draw) - this is memory efficient but limited by speed of data
>> medium access. This approach is used by GDAL, as well as old
>> ArcView 3.x.
>> 
>> BTW, have you tried to implement simplifications out of OGR?
> 
> I'm currently using this approach. The code is in libral. It in fact 
> gets the data from OGR and converts it into another structure before 
> rendering (laziness on my part, it was easier that way to implement
> the polygon rendering). I have not profiled the code extensively, it
> could indeed be worth while to put a simplification algorithm just on
> top of OGR.

Hmm, first I'd try to optimize by drawing directly from OGR structures.
I suppose that will boost it significantly.

>>> Often implementers show the incremental rendering so that users
>>> don't get bored while waiting, but it still is slow.
>> Yes, that's another possible improvement, also multi-threaded
>> rendering
> 
> That would be a good idea, but I'm not that familiar with 
> multi-threading and AFAIK OGR might not support multi-threading?

Yes, though the situation perhaps is not black or white here.
Generally, GDAL/OGR is not thread-safe, but drawing is a read-only
operation and some/many of GDAL/OGR drivers might behave correctly when
accessed from multiple threads.
However, some deeper analysis of thread-safety is required to judge.

Cheers
-- 
Mateusz Loskot
http://mateusz.loskot.net