From david.serret at eigendynamics.com Tue Oct 16 08:38:33 2018 From: david.serret at eigendynamics.com (David Serret Mayer) Date: Tue, 16 Oct 2018 17:38:33 +0200 Subject: [pgpointcloud] Uploading to pgpointcloud with pqxx Message-ID: Hi! I am looking to increase the performance of the uploading process with pgpointcloud and pqxx. After a bit of profiling and playing around I have seen that with the documentation on the README, by appending a huge string and then executing the sql code like this: ... pqxx::work work(conn); std::string command (" INSERT INTO patches (pa) SELECT PC_MakePatch (1, ARRAY[") for(auto& point : point_vector){ command += std::to_string(point.x); command += ", " command += std::to_string(point.y); command += ", " command += std::to_string(point.z); command += ", " command += std::to_string(point.i); command += ", " } command += "]));)");" work.exec(command); ... Is there a more efficient method. I feel doing everything from strings should not be faster. And the code above scales horribly with bigger point clouds. Also, I have tried to see what the guys at PDAL are doing and I think **of course** they are doing much better than me :P. Any code sample or read regarding this topic would be appreciated. Thank you in advance. -- *David Serret *- R&D Engineer at Eigen Dynamics SL -------------- next part -------------- An HTML attachment was scrubbed... URL: From pramsey at cleverelephant.ca Tue Oct 16 08:50:29 2018 From: pramsey at cleverelephant.ca (Paul Ramsey) Date: Tue, 16 Oct 2018 08:50:29 -0700 Subject: [pgpointcloud] Uploading to pgpointcloud with pqxx In-Reply-To: References: Message-ID: <7B5E3ADA-7B0E-415B-9ADB-1B049FFE2BBF@cleverelephant.ca> Directly writing into the “WKB” format is what the pdal folks do (I think, still). https://github.com/pgpointcloud/pointcloud/blob/master/lib/pc_api.h#L406 Surprised the format is not laid out in docs anywhere I can see. P > On Oct 16, 2018, at 8:38 AM, David Serret Mayer wrote: > > Hi! > > I am looking to increase the performance of the uploading process with pgpointcloud and pqxx. > > After a bit of profiling and playing around I have seen that with the documentation on the README, by appending a huge string and then executing the sql code like this: > > ... > pqxx::work work(conn); > std::string command (" > INSERT INTO patches (pa) SELECT PC_MakePatch (1, ARRAY[") > > for(auto& point : point_vector){ > command += std::to_string(point.x); > command += ", " > command += std::to_string(point.y); > command += ", " > command += std::to_string(point.z); > command += ", " > command += std::to_string(point.i); > command += ", " > } > command += "]));)");" > work.exec(command); > ... > > Is there a more efficient method. I feel doing everything from strings should not be > faster. And the code above scales horribly with bigger point clouds. > > > Also, I have tried to see what the guys at PDAL are doing and I think **of course** they are doing much better than me :P. Any code sample or read regarding this topic would be appreciated. > > > Thank you in advance. > > > > -- > David Serret - R&D Engineer at Eigen Dynamics SL > > _______________________________________________ > pgpointcloud mailing list > pgpointcloud at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/pgpointcloud -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.lemoine at oslandia.com Tue Oct 16 23:05:20 2018 From: eric.lemoine at oslandia.com (=?ISO-8859-1?Q?=C9ric?= Lemoine) Date: Wed, 17 Oct 2018 08:05:20 +0200 Subject: [pgpointcloud] Uploading to pgpointcloud with pqxx In-Reply-To: <7B5E3ADA-7B0E-415B-9ADB-1B049FFE2BBF@cleverelephant.ca> References: <7B5E3ADA-7B0E-415B-9ADB-1B049FFE2BBF@cleverelephant.ca> Message-ID: <92a07b6bcffea170c1ffba9f366fd96e52857750.camel@oslandia.com> On Tue, 2018-10-16 at 08:50 -0700, Paul Ramsey wrote: > Directly writing into the “WKB” format is what the pdal folks do (I > think, still). That's correct. > > > https://github.com/pgpointcloud/pointcloud/blob/master/lib/pc_api.h#L > 406 > > Surprised the format is not laid out in docs anywhere I can see. There's a "Binary Formats" section that may be useful in the README: https://github.com/pgpointcloud/pointcloud#binary-formats PDAL's PgWriter code may help as well: https://github.com/PDAL/PDAL/blob/0c71a7df07ccd55602bcad7b7462ef3a98ab317d/plugins/pgpointcloud/io/PgWriter.cpp#L411-L476 -- Éric Lemoine Oslandia +33 1 86 95 95 55 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part URL: