[GRASS-dev] Debugging, parallelism, etc.

[Even Rouault]

Hi,

just wanted to point that if you are interested in a "framework" for 
submit jobs to a thread pool, I can point to

https://github.com/uclouvain/openjpeg/blob/master/src/lib/openjp2/thread.h

https://github.com/uclouvain/openjpeg/blob/master/src/lib/openjp2/thread.c

which is a port in C I've done from the equivalent C++ code of GDAL 
(https://github.com/OSGeo/gdal/blob/master/port/cpl_worker_thread_pool.h).

It has a pthread and Win32 implementation. It could be easily extracted 
from libopenjp2 (pending a opj_ -> grass_ renaming to avoid conflicts if 
both are combined)

The high level API is the opj_thread_pool_* one.

Probably not super fancy, but serves my need well. The user is 
responsible for selecting the number of threads and splitting the work 
load in jobs that are queued to the thread pool and consumed by the 
threads as soon as they are no longer busy.

Even


Le 14/10/2022 à 11:02, Moritz Lennert a écrit :
> Am 09.10.2022 20:45 schrieb Brad ReDacted:
>> Those variables would be...? Is this documented somewhere? If not, can
>> we get it documented?
>
> https://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly 
>
>
>>
>> OpenMP can do far more than just loop "unrolling", these days. Tasking
>> sections to run concurrent is also quite trivial. It can also offload
>> to GPU, etc. Check out the v4.5+ spec. It's pretty impressive. I
>> believe it can do most of what pthreads does, but you certainly lose
>> control of implementation details. Some compilers have an omp library
>> while others convert to pthreads.
>>
>> I do find myself rewriting algorithms so that OpenMP can handle them.
>> It doesn't seem to handle nested loops with breaks very well and I'm
>> not entirely sure why.
>>
>> On 10/9/2022 10:43 AM, William Hargrove wrote:
>>> Can still run GRASS outside the shell by setting all of the 
>>> environment variables appropriately ...
>>>
>>> OpenMP just works by "unrolling" all of the determinate loops, i.e., 
>>> the ones that iterate a fixed number of times.  No speedups to 
>>> anything else.
>>>
>>> Speedup from OpenMP will be limited, depending on the number of 
>>> determinate loops present, and how much of the load they represent.
>>>
>>> pthreads are totally flexible, but the programmer has to specify 
>>> everything, very carefully ...
>>>
>>> But pthreads can speed up lots of stuff outside of determinate loops 
>>> ...
>>>
>>>
>>> HTH,
>>>
>>> Bill H.
>>>
>>>
>>> On 10/9/2022 12:37 PM, Brad ReDacted wrote:
>>>> Hello,
>>>>
>>>> I'm working on adding parallelism to modules, but debugging is 
>>>> turning out to be a logistical nightmare:
>>>>
>>>> Why do I not get any reporting from GCC option 
>>>> '-fsanitize=address|thread"?
>>>>
>>>> I am also having trouble getting the profiler to work properly 
>>>> inside GRASS (I assume due to shell?). The gmon.out file produced 
>>>> has no usable data.
>>>>
>>>> OpenMP is extremely poorly supported by most tools. valgrind with 
>>>> helgrind reports a lot of nonsense. I can't seem to get the Intel 
>>>> linux tools to work properly, either.
>>>>
>>>> BTW, we are supporting both pthreads and OpenMP. While this isn't 
>>>> an issue in most cases, there can be races and deadlocks if not 
>>>> handled properly. Pthreads aren't entirely portable. OpenMP is. 
>>>> However, pthreads gives us a more control. May I suggest using 
>>>> OpenMP for most modules and reserve Pthreads to libraries, etc? Or 
>>>> should we start moving away from pthreads?
>>>>
>>>> Any suggestions would be greatly appreciated!
>>>>
>>>>
>>>
>> -- 
>> Best Regards,
>> -Brad
>>
>> _______________________________________________
>> grass-dev mailing list
>> grass-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/grass-dev
>
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20221014/a30b41c2/attachment.htm>