[pdal] Fwd: [Boost-users] [interprocess] Reading huge files
Mateusz Loskot
mateusz at loskot.net
Fri Oct 11 06:46:25 PDT 2013
I think C++ for LAS/LiDAR craftsmen may want to monitor this
potentially interesting thread on Boost-users
---------- Forwarded message ----------
From: Sensei <senseiwa at gmail.com>
Date: 11 October 2013 13:55
Subject: [Boost-users] [interprocess] Reading huge files
To: boost-users at lists.boost.org
Dear all,
I am new to boost memory mapping, so this question might look simplistic.
I need to read huge amounts of data (for instance, a 20GB file), and
since memory mapping is quite fast, I was going to use it. However, I
don't know what it would be faster when, due to memory constraints, I
need to partition the file into regions. Moreover, I should treat the
file as a string (I need to perform string operations).
What I'm trying now is just to read the entire file:
boost::interprocess::file_mapping mmap(input_filename.c_str(),
boost::interprocess::read_only);
boost::interprocess::mapped_region map(mmap,
boost::interprocess::read_only);
std::size_t l = map.get_size(), tot_read = 0;
void *ptr = map.get_address();
while (tot_read < l)
{
register std::size_t x = std::min(l - tot_read,
static_cast<std::size_t>(prealloc));
std::copy_n(static_cast<char*>(ptr) + tot_read, x, line.begin());
// Do something here...
tot_read += x;
}
So, when the file is huge, do I need to create a mapped_region inside
the loop? I didn't see anywhere in the documentation the possibility
to move the mapped region.
Another side-question, if you don't mind. I'm not sure that what I'm
doing is efficient, especially the need to copy from the region to a
string. If you have suggestions, I'm more than happy to hear these.
Thanks & Cheers!
_______________________________________________
Boost-users mailing list
Boost-users at lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users
--
Mateusz Loskot, http://mateusz.loskot.net
More information about the pdal
mailing list