[Liblas-devel] tackling the root of inefficiency, with Parallel Python, LAStools

[Aaron Reyna]

OK, sorry for the cross post with the LAStools mailing list, but I thought
this was kind of important. Not directly liblas, but close enough.

All,

been so busy as of late with a new job that I haven't had time to post this
simple combination of parallel python and LAStools. Wrote it on a plane to
germany, so it only is set to SMP, but it could easily be dumped across a
system with several hundred nodes. Ah, I love python.


import pp, time

# Import system modules
import sys, string, os

def text_blaster(s):
    com = "C:/OSGeo4W/bin/las2txt.exe -i C:/projects/SB/input/SBEX_41.las -o
C:/projects/SB/output/" + s
    app_run = str(com)
    z = os.system(app_run)

my_list = ["SBEX_41.txt", "SBEX_41_2.txt", "SBEX_41_3.txt", "SBEX_41_4.txt",
"SBEX_41_5.txt", "SBEX_41_6.txt", "SBEX_41_7.txt", "SBEX_41_8.txt",]


print """Usage: python test_SMP.py [ncpus]
    [ncpus] - the number of workers to run in parallel,
    if omitted it will be set to the number of processors in the system
"""

# tuple of all parallel python servers to connect with
ppservers = ("*",)

if len(sys.argv) > 1:
    ncpus = int(sys.argv[1])
    # Creates jobserver with ncpus workers
    job_server = pp.Server(ncpus, ppservers=ppservers)
else:
    # Creates jobserver with automatically detected number of workers
    job_server = pp.Server(ppservers=ppservers)
start_time = time.time()

print "Starting pp with", job_server.get_ncpus(), "workers"
for process in my_list:
    job1 = job_server.submit(text_blaster, (process,))

print "Time elapsed: ", time.time() - start_time, "s"

result = job1()
#print "CLUSTER TEST is", result


# Submitted jobs and retrieved results

print "Time elapsed: ", time.time() - start_time, "s"
job_server.print_stats()

RESULTS:-------------------------------------------------------------------------------------------------------------------------------

Starting pp with 1 workers
Time elapsed:  0.00399994850159 s
Time elapsed:  60.1719999313 s
Job execution statistics:
 job count | % of all jobs | job time sum | time per job | job server
         8 |        100.00 |      60.1650 |     7.520625 | local
Time elapsed since server creation 60.1719999313
------------------------------------------------------------------------------------------------------------------------------------------------
Starting pp with 4 workers
Time elapsed:  0.00999999046326 s
Time elapsed:  27.8439998627 s
Job execution statistics:
 job count | % of all jobs | job time sum | time per job | job server
         8 |        100.00 |      82.2060 |    10.275750 | local
Time elapsed since server creation 27.8459999561
------------------------------------------------------------------------------------------------------------------------------------------------


Used one las file, 500 meters by 500 meters, 30pts per square meter. Time
was cut in half. Well, less than half.

Cheers,

Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/liblas-devel/attachments/20110416/e97ed956/attachment.html