[Liblas-devel] tackling the root of inefficiency, with Parallel Python, LAStools

[tisham at eleceng.adelaide.edu.au]

Hi Aaron,

Python does make it simple but you are not really using interprocess
communication, I wrote some MPI based code to perform spatial filtering on
Lidar data. It may be of interest to you for performance testing -
https://code.google.com/p/lasmpi/

Cheers,

Tisham.

> OK, sorry for the cross post with the LAStools mailing list, but I thought
> this was kind of important. Not directly liblas, but close enough.
>
> All,
>
> been so busy as of late with a new job that I haven't had time to post
> this
> simple combination of parallel python and LAStools. Wrote it on a plane to
> germany, so it only is set to SMP, but it could easily be dumped across a
> system with several hundred nodes. Ah, I love python.
>
>
> import pp, time
>
> # Import system modules
> import sys, string, os
>
> def text_blaster(s):
>     com = "C:/OSGeo4W/bin/las2txt.exe -i C:/projects/SB/input/SBEX_41.las
> -o
> C:/projects/SB/output/" + s
>     app_run = str(com)
>     z = os.system(app_run)
>
> my_list = ["SBEX_41.txt", "SBEX_41_2.txt", "SBEX_41_3.txt",
> "SBEX_41_4.txt",
> "SBEX_41_5.txt", "SBEX_41_6.txt", "SBEX_41_7.txt", "SBEX_41_8.txt",]
>
>
> print """Usage: python test_SMP.py [ncpus]
>     [ncpus] - the number of workers to run in parallel,
>     if omitted it will be set to the number of processors in the system
> """
>
> # tuple of all parallel python servers to connect with
> ppservers = ("*",)
>
> if len(sys.argv) > 1:
>     ncpus = int(sys.argv[1])
>     # Creates jobserver with ncpus workers
>     job_server = pp.Server(ncpus, ppservers=ppservers)
> else:
>     # Creates jobserver with automatically detected number of workers
>     job_server = pp.Server(ppservers=ppservers)
> start_time = time.time()
>
> print "Starting pp with", job_server.get_ncpus(), "workers"
> for process in my_list:
>     job1 = job_server.submit(text_blaster, (process,))
>
> print "Time elapsed: ", time.time() - start_time, "s"
>
> result = job1()
> #print "CLUSTER TEST is", result
>
>
> # Submitted jobs and retrieved results
>
> print "Time elapsed: ", time.time() - start_time, "s"
> job_server.print_stats()
>
> RESULTS:-------------------------------------------------------------------------------------------------------------------------------
>
> Starting pp with 1 workers
> Time elapsed:  0.00399994850159 s
> Time elapsed:  60.1719999313 s
> Job execution statistics:
>  job count | % of all jobs | job time sum | time per job | job server
>          8 |        100.00 |      60.1650 |     7.520625 | local
> Time elapsed since server creation 60.1719999313
> ------------------------------------------------------------------------------------------------------------------------------------------------
> Starting pp with 4 workers
> Time elapsed:  0.00999999046326 s
> Time elapsed:  27.8439998627 s
> Job execution statistics:
>  job count | % of all jobs | job time sum | time per job | job server
>          8 |        100.00 |      82.2060 |    10.275750 | local
> Time elapsed since server creation 27.8459999561
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> Used one las file, 500 meters by 500 meters, 30pts per square meter. Time
> was cut in half. Well, less than half.
>
> Cheers,
>
> Aaron
> _______________________________________________
> Liblas-devel mailing list
> Liblas-devel at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/liblas-devel
>