[GRASS-dev] Re: About the vector changes
Helena Mitasova
hmitaso at unity.ncsu.edu
Sat Aug 8 15:11:00 EDT 2009
I did not have time to respond to this line of discussion as I am
trying to get my class material ready,
but lib/rst also needs to be updated with offsets set to off_t and
fseek to G_fseek
in interp2d.c and write2d.c - see below. This should be done only in
grass7.
I think GRASS64 / 65 rst version should stay as it is - with the
recent fixes
it already should support raster output up to around 30,000x30,000
computed from
tens of millions of points on a higher end PC so if anybody wants to
do anything bigger
I would suggest to use grass7. Also the recent bug fixes in
v.surf.rst done in grass65
need to be ported to grass7 - it is not as straightforward as I
thought - there are too many differences
between grass6 and grass7 already (perhaps time to finish GRASS64 and
release it -
RC1 has been released in December 2008),
Helena
here is my exchange with Glynn on offset topic :
> For 38300x30200 raster when writing to temp file, the program says
> Cannot fseek elev offset2=-438164932
>
> due to this in write2d.c:
> if (fseek(params->Tmp_fd_z, (long)offset2, 0) == -1) {
> fprintf(stderr, "Cannot fseek elev offset2=%d\n", (int)offset2);
>
> offset2 is computed in interp2d.c as follows:
>
> offset1 is number of columns
> offset = offset1 * (k - 1); is rows offset, k=1,nrows so this can be
> 38300x30200
> offset2 = (offset + ngstc - 1) * sizeof(FCELL) where ngstc is
> starting column
>
> for example:
>
> here it does not give error but offset2 is apparently wrong
> offset=1077052800, offset1=30200, k=35665
> offset=1077052800, offset2=13245556, ngstc=414, fcell=4
>
> here offset2 is negative so it gives an error:
> offset=1070318200, offset1=30200, k=35442
> offset=1070318200, offset2=-13694496, ngstc=1, fcell=4
> Cannot fseek elev offset2=-13694496
>
> I am on 64bit machine compiled with large file support
> and I put this into Makefile (according the the advice on the devlist)
>
> ifneq ($(USE_LARGEFILES),)
> EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64 $(VECT_CFLAGS)
> endif
>
> hoping that would take care of it, but apparently that is not enough.
> all offsets above are defined as int.
>
> So how to fix this?
> Should I just replace offset2 with off_t as suggested in the dev list?
> Do I need to do anything with fseek?
> or am I completely misunderstanding this issue?
>
LFS doesn't work with fseek(); but LFS does nothing on a 64-bit system
anyhow.
The main problem is that offset1 and offset2 are "int"s, and they need
to be either "long" or "off_t". Also, when you calculate an offset by
multiplying a row number by the row width, one of the operands must be
cast to "long" (or "off_t") *before* the multiply, e.g.:
off_t offset = (off_t) row * row_size;
Casting the result of a multiplication won't work; the value will
already have been truncated.
For LFS (files >2GiB on 32-bit systems), the same issues apply,
except:
1. offsets need to be "off_t" rather than "long".
2. fseek() and ftell() won't work, as they use "long" rather than
"off_t". Use G_fseek() and G_ftell() instead; these use the fseeko()
and ftello() functions on platforms where they are available.
Helena Mitasova
Associate Professor
Department of Marine, Earth
and Atmospheric Sciences
1125 Jordan Hall, Campus Box 8208
North Carolina State University
Raleigh NC 27695-8208
http://skagit.meas.ncsu.edu/~helena/
On Aug 8, 2009, at 2:34 PM, Markus Neteler wrote:
> On Sat, Aug 8, 2009 at 7:03 PM, Glynn
> Clements<glynn at gclements.plus.com> wrote:
>>
>> Markus Metz wrote:
>>
>>>>> Yes, if you want to work with vectors not in the current mapset
>>>>> using
>>>>> grass7, you have to rebuild topology for all vectors in the
>>>>> corresponding mapset first.
>>>>>
>>>>
>>>> So it's intentional that GRASS7 is no longer compatible with
>>>> previous
>>>> vector maps?
>>>
>>> Yes, for two reasons. GRASS7 expects now a sidx file while GRASS6
>>> doesn't, and the last GRASS version writing a sidx file (I think
>>> GRASS57) wrote it differently to the new format. I have now
>>> changed the
>>> format numbers and set forward/backward compatibility info.
>>> GRASS6 can
>>> open GRASS7 vectors, but GRASS7 can't open GRASS6 or GRASS5 vectors,
>>> rebuilding topology is required. IOW, I broke backwards
>>> compatibility
>>> which is against GRASS development policy I guess.
>>
>> Breaking backwards compatibility is allowed in 7.0. We'll probably be
>> breaking raster compatibility at some point (but in a far more
>> fundamental way; old rasters won't be recognised at all).
>>
>> Something this fundamental should have been announced (maybe it was
>> and I missed it). Also, the error message is misleading.
>>
>>> My argument is that
>>> enabling backwards compatibility would be about the same effort like
>>> rebuilding topology. I could write a module that converts any
>>> existing
>>> sidx file to the new format or builds the spatial index from topo
>>> if no
>>> sidx file is found, then writes a new sidx file in the new
>>> format. That
>>> would avoid rebuilding the core topology and category index.
>>
>> Maybe the sidx file should have been renamed?
>>
>>>> That isn't necessarily a problem, but I must have missed the
>>>> announcement.
>>> Was hidden here:
>>> http://lists.osgeo.org/pipermail/grass-dev/2009-July/045098.html
>>
>> Right; I didn't get the significance of that at the time.
>>
>>>> PERMANENT/vector/fields contains:
>>>>
>>>> -rw-r--r-- 1 root root 4117 May 25 2004 cidx
>>>> -rw-r--r-- 1 root root 27905 May 25 2004 coor
>>>> -rw-r--r-- 1 root root 55 May 25 2004 dbln
>>>> -rw-r--r-- 1 root root 278 May 25 2004 head
>>>> -rw-r--r-- 1 root root 186 May 25 2004 hist
>>>> -rw-r--r-- 1 root root 24538 May 25 2004 sidx
>>>
>>> There is no sidx file for vectors in
>>> http://grass.osgeo.org/sampledata/spearfish_grass60data-0.3.tar.gz
>>
>> IOW: sidx was removed in 6.0 (existing sidx files were silently
>> ignored), then an entirely different sidx file was added recently,
>> and
>> the code is confused by the existence of a 5.7 sidx file, right?
>>
>> A different name might have been a better choice. Although the 5.7
>> datasets are old, they worked until a couple of weeks ago, so people
>> may still be using them.
>>
>> Error messages about LFS could cause people to waste effort
>> rebuilding
>> GRASS to no effect (we've seen something similar with people being
>> misdirected by the "helpful" information added to the "incompatible
>> library version" errors).
>>
>>> Ah, ok, you use the grass57 dataset. I have now changed the version
>>> number and compatibility info, you should now get reasonable
>>> warnings/errors with the spearfish57 dataset.
>>
>> Okay, I'll check it.
>>
>>>> Also, this needs to be checked for anything which uses VECT_CFLAGS,
>>>> which isn't limited to v.* modules (a few g.* and r.* modules use
>>>> vector functions).
>>>
>>> I fixed r.drain, v.in.dxf, v.vol.rst. AFAICT all other g.*, r.*
>>> and v.*
>>> modules are ok. Please let me know if I missed something.
>>
>> I don't know which modules need fixing and which don't. I was just
>> pointing out that e.g. g.{copy,remove,rename,list} use the vector
>> libraries (but don't use VECT_CFLAGS; is that a bug?), r.to.vect,
>> d.vect, etc use the libraries and VECT_CFLAGS, so it isn't just v.*
>> modules which need to be checked. It might just be v.* modules which
>> need to be fixed; I don't know.
>>
>>> I got carried away (replacing all fseek/ftell occurrences I could
>>> find
>>> with G_fseek/G_ftell, adjusting off_t as you showed above) and
>>> also made
>>> r3.in|out.v5d LFS-safe, but did not submit. Should I?
>>
>> Yes; ideally, we should use G_{fseek,ftell} everywhere.
>
> Patch attached.
>
> Note that it now fails to compile here:
>
> make[1]: Entering directory `/home/neteler/grass70/lib/rst/
> interp_float'
> gcc -I/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/include
> -I/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/include -g
> -Wall -Werror-implicit-function-declaration -fno-common -mtune=nocona
> -m64 -minline-all-stringops -fPIC -I/usr/local/include
> -D_FILE_OFFSET_BITS=64 -DPACKAGE=\""grasslibs"\"
> -I/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/include
> -I/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/include -o
> OBJ.x86_64-unknown-linux-gnu/output2d.o -c output2d.c
> output2d.c: In function ‘IL_output_2d’:
> output2d.c:153: error: void value not ignored as it ought to be
> output2d.c:174: error: void value not ignored as it ought to be
> output2d.c:190: error: void value not ignored as it ought to be
> output2d.c:206: error: void value not ignored as it ought to be
> output2d.c:222: error: void value not ignored as it ought to be
> output2d.c:238: error: void value not ignored as it ought to be
> make[1]: *** [OBJ.x86_64-unknown-linux-gnu/output2d.o] Error 1
>
> No clue how what could be wrong in the patch. Same problem in
> raster/r.in.xyz
>
> Feel free to recycle the patch.
>
> Markus
> <G_fseek_ftell.diff.gz>
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
More information about the grass-dev
mailing list