[GRASS-dev] [SoC] Parallelization of Raster and Vector libraries

[Jordan Neumeyer]

Opps, forgot to send this to mailing list as well...

On Tue, Mar 30, 2010 at 8:32 PM, Jordan Neumeyer <
jordan.neumeyer at mines.sdsmt.edu> wrote:

> Hi Glynn,
>
> On Tue, Mar 30, 2010 at 8:29 AM, Glynn Clements <glynn at gclements.plus.com>wrote:
>
>>
>> Jordan Neumeyer wrote:
>>
>> > Just kind of my thought process about how I would try to go about
>> > parallelizing a module.
>>
>> The main issue with parallelising raster input is that the library
>> keeps a copy of the current row's data, so that consecutive reads of
>> the same row (as happen when upsampling) don't re-read the data.
>>
>> For concurrent access to a single map, you would need to either keep
>> one row per thread, or abandon caching. Also, you would need to use
>> pread() rather than fseek()+read().
>>
>
> It sounds like you're talking about parallelism in I/O from a file or
> database. Neither of which is my intent or goal for this project. I will
> parallelize things after they have already been read into memory, and tasks
> are processor intensive. I wouldn't want parallelize any I/O, but if I were
> to optimize I/O. I would make all operations I/O asynchronous, which is can
> mimic parallelism in a sense. Queuing up the chunks of data and then
> processing them as resources become available.
>
>
>> It's more straightfoward to read multiple maps concurrently. In 7.0,
>> this case should be thread-safe.
>>
>> Alternatively, you could have one thread for reading, one for writing,
>> and multiple worker threads for the actual processing. However, unless
>> the processing is complex, I/O will be the bottleneck.
>>
>
> I/O is generally a bottleneck anyway. Something always tends to be waiting
> on another.
>
>
>> --
>>
>> Glynn Clements <glynn at gclements.plus.com>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/grass-dev/attachments/20100401/5b3be752/attachment-0001.html