[pdal] Fwd: [Boost-users] [interprocess] Reading huge files

Mateusz Loskot mateusz at loskot.net
Fri Oct 11 06:46:25 PDT 2013


I think C++ for LAS/LiDAR craftsmen may want to monitor this
potentially interesting thread on Boost-users



---------- Forwarded message ----------
From: Sensei <senseiwa at gmail.com>
Date: 11 October 2013 13:55
Subject: [Boost-users] [interprocess] Reading huge files
To: boost-users at lists.boost.org


Dear all,

I am new to boost memory mapping, so this question might look simplistic.

I need to read huge amounts of data (for instance, a 20GB file), and
since memory mapping is quite fast, I was going to use it. However, I
don't know what it would be faster when, due to memory constraints, I
need to partition the file into regions. Moreover, I should treat the
file as a string (I need to perform string operations).

What I'm trying now is just to read the entire file:

    boost::interprocess::file_mapping mmap(input_filename.c_str(),
boost::interprocess::read_only);
    boost::interprocess::mapped_region map(mmap,
boost::interprocess::read_only);

    std::size_t l = map.get_size(), tot_read = 0;

    void *ptr = map.get_address();

    while (tot_read < l)
    {
        register std::size_t x = std::min(l - tot_read,
static_cast<std::size_t>(prealloc));

        std::copy_n(static_cast<char*>(ptr) + tot_read, x, line.begin());

        // Do something here...

        tot_read += x;
    }


So, when the file is huge, do I need to create a mapped_region inside
the loop? I didn't see anywhere in the documentation the possibility
to move the mapped region.

Another side-question, if you don't mind. I'm not sure that what I'm
doing is efficient, especially the need to copy from the region to a
string. If you have suggestions, I'm more than happy to hear these.


Thanks & Cheers!




_______________________________________________
Boost-users mailing list
Boost-users at lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users


-- 
Mateusz  Loskot, http://mateusz.loskot.net


More information about the pdal mailing list