OK, sorry for the cross post with the LAStools mailing list, but I thought this was kind of important. Not directly liblas, but close enough.<br><br>All,<br><br>been so busy as of late with a new job that I haven't had time to post this simple combination of parallel python and LAStools. Wrote it on a plane to germany, so it only is set to SMP, but it could easily be dumped across a system with several hundred nodes. Ah, I love python.<br>
<br><br>import pp, time<br><br># Import system modules<br>import sys, string, os<br><br>def text_blaster(s):<br> com = "C:/OSGeo4W/bin/las2txt.exe -i C:/projects/SB/input/SBEX_41.las -o C:/projects/SB/output/" + s<br>
app_run = str(com)<br> z = os.system(app_run)<br><br>my_list = ["SBEX_41.txt", "SBEX_41_2.txt", "SBEX_41_3.txt", "SBEX_41_4.txt", "SBEX_41_5.txt", "SBEX_41_6.txt", "SBEX_41_7.txt", "SBEX_41_8.txt",]<br>
<br><br>print """Usage: python test_SMP.py [ncpus]<br> [ncpus] - the number of workers to run in parallel, <br> if omitted it will be set to the number of processors in the system<br>"""<br>
<br># tuple of all parallel python servers to connect with<br>ppservers = ("*",)<br><br>if len(sys.argv) > 1:<br> ncpus = int(sys.argv[1])<br> # Creates jobserver with ncpus workers<br> job_server = pp.Server(ncpus, ppservers=ppservers)<br>
else:<br> # Creates jobserver with automatically detected number of workers<br> job_server = pp.Server(ppservers=ppservers)<br>start_time = time.time()<br><br>print "Starting pp with", job_server.get_ncpus(), "workers"<br>
for process in my_list:<br> job1 = job_server.submit(text_blaster, (process,))<br><br>print "Time elapsed: ", time.time() - start_time, "s"<br><br>result = job1()<br>#print "CLUSTER TEST is", result<br>
<br><br># Submitted jobs and retrieved results<br><br>print "Time elapsed: ", time.time() - start_time, "s"<br>job_server.print_stats()<br><br>RESULTS:-------------------------------------------------------------------------------------------------------------------------------<br>
<br>Starting pp with 1 workers<br>Time elapsed: 0.00399994850159 s<br>Time elapsed: 60.1719999313 s<br>Job execution statistics:<br> job count | % of all jobs | job time sum | time per job | job server<br> 8 | 100.00 | 60.1650 | 7.520625 | local<br>
Time elapsed since server creation 60.1719999313<br>------------------------------------------------------------------------------------------------------------------------------------------------<br>Starting pp with 4 workers<br>
Time elapsed: 0.00999999046326 s<br>Time elapsed: 27.8439998627 s<br>Job execution statistics:<br> job count | % of all jobs | job time sum | time per job | job server<br> 8 | 100.00 | 82.2060 | 10.275750 | local<br>
Time elapsed since server creation 27.8459999561<br>------------------------------------------------------------------------------------------------------------------------------------------------<br><br><br>Used one las file, 500 meters by 500 meters, 30pts per square meter. Time was cut in half. Well, less than half.<br>
<br>Cheers,<br><br>Aaron<br>