[GRASS-user] ERROR: Bytes do not match file size with r.in.bin (but file size is correct!!)

[Ludovico Nicotina]

Hi and thank you both,

In my case the problem is not the fortran record markers, indeed the files are exactly the size they are supposed to be. Additionally I have doublechecked and found out that I'm actually running GRASS 6.4.2 so I'm wondering if the large file support is something that is embedded in that distribution or if it should have been explicitly included at installation time. Do you know if there is a way to check that and/or if it's possible to add that without repeating the entire installation?

Thanks again,

PS. Hamish I will probably switch to 6.4.3rc3 anyway, would this solve the problem automatically in your opinion?


From: kapo coulibaly [mailto:kmcoulib at gmail.com]
Sent: 09 May 2013 14:49
To: Hamish
Cc: GRASS user list; Ludovico Nicotina
Subject: Re: [GRASS-user] ERROR: Bytes do not match file size with r.in.bin (but file size is correct!!)

Hi Ludvico,
You also want to check how the binary was written in FORTRAN. depending on the compiler and/or the access type fortran can write extra bytes before and after every record (they are called record marker). In that particular case the file would be bigger than the information it is supposed to contain. It is explained here: http://paulbourke.net/dataformats/reading/
I copied the content of the site below in case the link doesn't make it.

Problem

Ever wanted to read binary files written by a FORTRAN program with a C/C++ program? Not such an unusual or unreasonable request but FORTRAN does some strange things ..... consider the following FORTRAN code, where "a" is a 3D array of 4 byte floating point values.

        open(60,file=filename,status='unknown',form='unformatted')

        write(60) nx,ny,nz

        do k = 1,nz

          do j = 1,ny

           write(60) (a(i,j,k),i=1,nx)

          enddo

        enddo

        close(60)

What you will end up with is not a file that is (4 * nx) * ny * nz + 12 bytes long as it would be for the equivalent in most (if not all) other languages! Instead it will be nz * ny * (4 * nx + 8) + 20 bytes long. Why?
Reason

Each time the FORTRAN write is issued a "record" is written, the record consists of a 4 byte header, then the data, then a trailer that matches the header. The 4 byte header and trailer consist of the number of bytes that will be written in the data section. So the following

        write(60) nx,ny,nz
gets written on the disk as follows where nx,ny,nz are each 4 bytes, the other numbers below are 2 byte integers written in decimal

        0 12 nx ny nz 0 12
The total length written is 20 bytes. Similarly, the line

        write(60) (a(i,j,k),i=1,nx)
gets written as follows assuming nx is 1024 and "a" is real*4

        10 0 a(1,j,k) a(2,j,k) .... a(1024,j,k) 10 0

The total length is 4104 bytes. Fortunately, once this is understood, it is a trivial to read the correct things in C/C++.

A consequence that is a bit shocking for many programmers is that the file created with the above code gives a file that is about 1/3 the size than one created with this code.

        open(60,file=filename,status='unknown',form='unformatted')

        write(60) nx,ny,nz

        do k = 1,nz

          do j = 1,ny

            do i = 1,nx

              write(60) a(i,j,k)

            enddo

          enddo

        enddo

        close(60)

In this case each element of a is written in one record and consumes 12 bytes for a total file size of nx * ny * nz * 12 + 20.




Hope it helps

On Wed, May 8, 2013 at 5:27 AM, Hamish <hamish_b at yahoo.com<mailto:hamish_b at yahoo.com>> wrote:
Ludovico wrote:
> Thank you for your answers. A few info on my system:
>
> I'm running on a computational node under linux with 64GB or
> RAM the machine architecture is a x86_64 and the kernel is
> also 64bit (running getconf LONG_BIT output is 64)
>
> The version of GRASS I am running is 6.4.1
You'll have to upgrade to a newer version. The fix for r.in.bin
was added just a few days after the release of 6.4.1, which was
two years ago.

I'd suggest 6.4.3rc3, get in early and help us test the upcoming
release. :)


Hamish
_______________________________________________
grass-user mailing list
grass-user at lists.osgeo.org<mailto:grass-user at lists.osgeo.org>
http://lists.osgeo.org/mailman/listinfo/grass-user


________________________________
This message and any attachments contain information that may be RMS Inc. confidential and/or privileged. If you are not the intended recipient (or authorized to receive for the intended recipient), and have received this message in error, any use, disclosure or distribution is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to the e-mail and permanently deleting the message from your computer and/or storage system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20130509/2f6e04d1/attachment-0001.html>