<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="Times New Roman, Times, serif">I'm working on multi-core
VMss in a Cloud environment, that access their data on a central
dataserver via NFS. Parallellizing jobs for different map sheets
gives huge accelerations for C-programs like gdaladdo, but there
seems to be a problem with Python-based programs like rgb2pct.py.
Consider the following:<br>
<br>
(<br>
rgb2pct.py file1.tif file1_256.tif<br>
gdaladdo file1_256.tif 2 4 8 16<br>
)& </font><br>
<font face="Times New Roman, Times, serif">(<br>
rgb2pct.py file1.tif file1_256.tif<br>
gdaladdo file1_256.tif 2 4 8 16<br>
)& <br>
</font>.. <font face="Times New Roman, Times, serif">etc, for all
available cores<br>
wait<br>
<br>
When running this on a 16-core VM I see first 16 python processes,
each with CPU-loads around 20% for each processor, and then 16
gdaladdo processes, with CPU-loads around 95%. When I replace the
tif-input-files for rgb2pct.py by equivalent jpg-files, the loads
for the 16 rgb2pct.py processes increase to about 80% and the
overall computing time more than halves.<br>
<br>
So my impression is that one Python I/O process blocks all
others. I have read something about Python's GIL (Global
Interpreter Lock,
<a class="moz-txt-link-freetext" href="http://docs.python.org/faq/library#can-t-we-get-rid-of-the-global-interpreter-lock">http://docs.python.org/faq/library#can-t-we-get-rid-of-the-global-interpreter-lock</a>)
and the multi-processing module, but I don't see an easy way to
implement this for my setup. Does anyone have a simple solution
for this problem?<br>
<br>
Jan <br>
</font><br>
</body>
</html>