[pdal] Intermittent crash calling LasReader::execute( PointTable )
Peder Axensten
Peder.Axensten at slu.se
Sat May 1 06:28:24 PDT 2021
I followed your advice and found a couple of issues that I’m happy I identified and fixed, one of which was related to the problem I had. In OS X all now compiles and runs without any error or warning, even with saitizers on. Also in Ubuntu under Docker if I compile without sanitizers. So thank you Andrew for the advice, my problem is solved.
However, if I compile in Ubuntu with sanitizers on there are things flagged that seem unrelated to my code.
==9474==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 15 byte(s) in 1 object(s) allocated from:
#0 0x7f8275f9f9f7 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:454
#1 0x7f82741c2fb7 in __static_initialization_and_destruction_0 /tmp/PDAL/vendor/kazhdan/Ply.h:206
SUMMARY: AddressSanitizer: 15 byte(s) leaked in 1 allocation(s).
Also, reading a file that it reads with sanitizers off, I get an exception from pdal:
'Invalid VLR offset - exceeds file size’.
I realise that it might all be a bit vague, but does anyone understand the reasons for the above?
Otherwise, I don’t plan to pursue this any longer.
Best regards,
Peder Axensten
Research engineer
Remote Sensing
Department of Forest Resource Management
Swedish University of Agricultural Sciences
SE-901 83 Umeå
Visiting address: Skogsmarksgränd
Phone: +46 90 786 85 00
peder.axensten at slu.se, www.slu.se/srh
The Department of Forest Resource Management is environmentally certified in accordance with ISO 14001.
> On 27 Apr 2021, at 13:47, Andrew Bell <andrew.bell.ia at gmail.com> wrote:
>
> If your code runs on OSX but fails on Linux, it's likely you have an issue accessing invalid memory. You should run valgrind or compile with -fsanitize=address to help you find your error. There are also many working example and test programs as part of the PDAL distribution that may help.
>
> On Tue, Apr 27, 2021 at 6:46 AM Peder Axensten <Peder.Axensten at slu.se> wrote:
> Hi,
>
> I’m in the process of changing from liblas to pdal and all our tools but one have now – with great help from this list! – been converted into pdal filters. The remaining tool reads all las/laz-files in a given directory. I have not yet figured out a good way to make it fit into the pdal filter paradigm, so for now I’ll keep the tool, but switch the file reading from liblas to pdal. I found information on how to do this at https://pdal.io/api/transition/index.html. I put the specific code into the class Point_file_reader, as seen below. It runs perfectly on OS X, but we process data in Ubuntu under Docker and this crashes under certain circumstances...
>
> If I simply construct a Point_file_reader object all works fine also in Ubuntu, but used in the processing environment it crashes with SIGSEGV - Segmentation violation signal. I really can not see how the processing environment could affect this isolated code as there are no apparent dependencies apart from the file path, but I’m obviously wrong. My next step will be to simplify until the crash goes away, but as this is rather cumbersome I first want to ask this list if someone have had similar experiences?
>
> The crash happens in laszip, called by pdal (as seen further below). In OS X I use clang, in Ubuntu I use gcc. The versions of pdal and laszip are as of last week.
>
> Any insights or ideas will be much appreciated!
>
> ==============================
> #pragma once
>
> #include <filesystem>
> #include <memory>
> #include <pdal/PointTable.hpp>
> #include <pdal/PointView.hpp>
> #include <pdal/io/LasHeader.hpp>
> #include <pdal/io/LasReader.hpp>
> #include <pdal/Options.hpp>
>
> namespace pax {
>
> /// Open an aprs las file (possibly compressed).
> /** Code from https://pdal.io/api/transition/index.html**/
> class Point_file_reader {
> pdal::LasReaderm_reader{};
> pdal::PointTablem_table{};
> pdal::PointViewSetm_point_view_set{};
> pdal::PointViewPtrm_point_view{};
>
> public:
> using point_type = pdal::PointRef;
> using header_type = pdal::LasHeader;
> using Id = pdal::Dimension::Id;
>
> Point_file_reader( const std::filesystem::path & file_ ) {
> pdal::Optionoption( "filename", file_.native() );
> pdal::Optionsoptions;
> options.add( option );
> m_reader.setOptions( options );
> m_reader.prepare( m_table );
>
> // Next line in Ubuntu 20.10 under Docker: SIGSEGV - Segmentation violation signal
> m_point_view_set = pdal::PointViewSet{ m_reader.execute( m_table ) };
>
> m_point_view = pdal::PointViewPtr{ *m_point_view_set.begin() };
> }
>
> const header_type & header() const {return m_reader.header();}
> auto begin() const {return m_point_view->begin();}
> auto end() const {return m_point_view->end();}
> bool has_dim( const Id dim_)const {return m_point_view->hasDim( dim_ );}
> };
>
> }//namespace pax
> ==============================
>
> I’ve installed gdb in the Docker environment, but I’m not very familiar with it. This is what I get:
>
> ==============================
> (gdb) run
> Starting program: /pax/build/Linux/point-cloud
> warning: Error disabling address space randomization: Operation not permitted
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> [doctest] run with "--help" for options
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007f55095c5cbd in std::istream::read(char*, long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
> (gdb) up
> #1 0x00007f5509118ee3 in ByteStreamInIstream::getBytes(unsigned char*, unsigned int) () from /usr/local/lib/liblaszip.so.6
> (gdb) up
> #2 0x00007f5506055b73 in laszip_read_header (laszip_dll=0x557806355e30, is_compressed=0x7ffd6e735354) at /tmp/LASzip/src/laszip_dll.cpp:3454
> 3454 try { laszip_dll->streamin->getBytes((U8*)file_signature, 4); } catch(...)
> (gdb) up
> #3 0x00007f5506059c44 in laszip_open_reader_stream (pointer=0x557806355e30, stream=..., is_compressed=is_compressed at entry=0x7ffd6e735354)
> at /tmp/LASzip/src/laszip_dll.cpp:4810
> 4810 return laszip_read_header(laszip_dll, is_compressed);
> (gdb) up
> #4 0x00007f5509a817d0 in pdal::LasReader::ready (this=0x7ffd6e7359c0, table=...) at /usr/include/c++/10/bits/unique_ptr.h:421
> 421 get() const noexcept
> (gdb) up
> #5 0x00007f5509bb75c9 in pdal::Stage::execute (this=0x7ffd6e735ac0, table=..., views=std::set with 1 element = {...}) at /usr/include/c++/10/ext/atomicity.h:84
> 84 return __exchange_and_add(__mem, __val);
> (gdb) up
> #6 0x00007f5509bb7d7d in pdal::Stage::execute (this=<optimized out>, table=...) at /tmp/PDAL/pdal/Stage.cpp:235
> 235 outViews = si.m_stage->execute(table, inViews);
> (gdb) up
> #7 0x0000557805cad5c8 in pax::Process_plots<pax::Aggregator_set<pax::Filtered_summary<float, 0u, pax::filter::Pass_all_filter>, pax::Filtered_summary<float, 4u, pax::filter::All_ge_level_filter>, pax::Filtered_summary<float, 0u, pax::filter::First_return_filter>, pax::Filtered_summary<float, 0u, pax::filter::First_return_ge_level_filter>, pax::Filtered_ordered<float, pax::filter::All_ge_level_filter, std::allocator<float> > > >::process_file (this=this at entry=0x7ffd6e735e80,
> file_=filesystem::path "../../doctests-data/point-cloud/original/09D037_68200_5175_25.laz" = {...}) at /usr/include/c++/10/ostream:517
> 517 operator<<(basic_ostream<char, _Traits>& __out, char __c)
> (gdb) up
> #8 0x0000557805caf12e in pax::Process_plots<pax::Aggregator_set<pax::Filtered_summary<float, 0u, pax::filter::Pass_all_filter>, pax::Filtered_summary<float, 4u, pax::filter::All_ge_level_filter>, pax::Filtered_summary<float, 0u, pax::filter::First_return_filter>, pax::Filtered_summary<float, 0u, pax::filter::First_return_ge_level_filter>, pax::Filtered_ordered<float, pax::filter::All_ge_level_filter, std::allocator<float> > > >::process_directory (this=this at entry=0x7ffd6e735e80,
> pc_directory_=filesystem::path "../../doctests-data/point-cloud/original" = {...}) at /usr/include/c++/10/bits/fs_dir.h:161
> 161 operator const filesystem::path& () const noexcept { return _M_path; }
> (gdb) up
> #9 0x0000557805caf5c5 in pax::process_plots<pax::Aggregator_set<pax::Filtered_summary<float, 0u, pax::filter::Pass_all_filter>, pax::Filtered_summary<float, 4u, pax::filter::All_ge_level_filter>, pax::Filtered_summary<float, 0u, pax::filter::First_return_filter>, pax::Filtered_summary<float, 0u, pax::filter::First_return_ge_level_filter>, pax::Filtered_ordered<float, pax::filter::All_ge_level_filter, std::allocator<float> > > > (pc_count_=0, do_progress_=<optimized out>,
> nilsson_=<optimized out>, pc_directory_=filesystem::path "../../doctests-data/point-cloud/original" = {...},
> plots_dest_=filesystem::path "/tmp/plots_result.csv" = {...}, plots_source_=filesystem::path "/tmp/plots_result.csv" = {...}, accessor_set_=...)
> at /pax/include/point-cloud/process/plot-metrics-implementation.hpp:190
> 190Json_value process_plots(
> (gdb) up
> #10 pax::process_plots<std::filesystem::__cxx11::path const&, std::filesystem::__cxx11::path const&, std::filesystem::__cxx11::path const&, double, bool> (
> accessor_ref_set_=...) at /pax/include/point-cloud/process/plot-metrics-implementation.hpp:219
> 219return process_plots( std::move( accessor_set ), std::forward< Args >( args_ )... );
> (gdb) up
> #11 0x0000557805c7f877 in pax::_DOCTEST_ANON_FUNC_4 () at /usr/include/c++/10/bits/stl_iterator.h:953
> 953 __normal_iterator(const _Iterator& __i) _GLIBCXX_NOEXCEPT
> (gdb) up
> #12 0x0000557805cc89b8 in doctest::Context::run (this=0x7ffd6e736980) at /pax/include/3rd-party/doctest/doctest.h:6486
> 6486 tc.m_test();
> (gdb) up
> #13 0x0000557805bf059c in main (argc=1, argv=0x7ffd6e736a98) at /pax/doctests-src/main.doctest.cpp:42
> 42const auto test_result = context.run();
> (gdb) up
> Initial frame selected; you cannot go up.
> ==============================
>
>
> Best regards,
>
> Peder Axensten
> Research engineer
>
> Remote Sensing
> Department of Forest Resource Management
> Swedish University of Agricultural Sciences
> SE-901 83 Umeå
> Visiting address: Skogsmarksgränd
> Phone: +46 90 786 85 00
> peder.axensten at slu.se, www.slu.se/srh
>
> The Department of Forest Resource Management is environmentally certified in accordance with ISO 14001.
>
> ---
> När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <https://www.slu.se/om-slu/kontakta-slu/personuppgifter/>
> E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <https://www.slu.se/en/about-slu/contact-slu/personal-data/>
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pdal
>
>
> --
> Andrew Bell
> andrew.bell.ia at gmail.com
---
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <https://www.slu.se/om-slu/kontakta-slu/personuppgifter/>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <https://www.slu.se/en/about-slu/contact-slu/personal-data/>
More information about the pdal
mailing list