[GRASS-dev] Allocating memory for a huge array

[Paul Kelly]

On Fri, 23 Mar 2007, Damiano Triglione wrote:

> Dear Paul and Glenn,
> yes: in order to be quick and short, I badly explained myself.
> Your advices are all interesting and correct, but I need to be more precise 
> about my case:
>
> I do not want a Huge Array, but a standard array (S_Array) whose values are 
> indices to a Huge Array (H_Array)!
> Besides, 2^32 is not the value I typed in my source file, but it was an 
> expression just to let you understand the value 4 billions.
>
> Sorry for my faults!
>
> So, my intention is, actually:
>
> unsigned int * S_Array;
> double * H_Array;
> (...)
> S_Array=(unsigned int *)G_calloc(100, sizeof(unsigned int));

Why not:
size_t *S_Array;
...
S_Array=(size_t *)G_calloc(100, sizeof(size_t));

?

As I see it you are relying on the fact that sizeof(unsigned int) is the 
same as sizeof(size_t), which won't be true on all platforms 
(specifically: on 64-bit size_t will be 64 bits wide whereas int/unsigned 
int will be only 32 in most cases).

> H_Array=(double *)G_calloc(1L<<32, sizeof(double));
>
>
> Does it make sense, now?
> Damiano
>
>
> ----- Original Message ----- From: "Glynn Clements" 
> <glynn at gclements.plus.com>
> To: "Damiano Triglione" <damiano.triglione at polimi.it>
> Cc: <grass-dev at grass.itc.it>
> Sent: Friday, March 23, 2007 6:23 AM
> Subject: Re: [GRASS-dev] Allocating memory for a huge array
>
>
>> 
>> Damiano Triglione wrote:
>> 
>>> in a Grass module I am developing, I would like to allocate a huge array. 
>>> I thought to use something like:
>>> 
>>> unsigned int * H_Array;
>>> (...)
>>> H_Array=(unsigned int *)G_calloc(2^32, sizeof(unsigned int));
>>> 
>>> because in this way (unsigned int) with 32 bits I can manage 4 billions 
>>> cells, instead of 2 billions related to 32 bits as (traditional signed) 
>>> int.
>>> At compile-time, I receive a warning:
>>>  warning: passing arg 1 of `G_calloc' as signed due to prototype
>>> 
>>> Thus, I realize that I am not succeeding in what I want!
>>> Is there a different G_calloc function that provides a prototype suitable 
>>> for my needs?
>> 
>> G_calloc() takes size_t arguments, and isn't the problem here.
>> 
>> There are two distinct problems with the line:
>> 
>> H_Array=(unsigned int *)G_calloc(2^32, sizeof(unsigned int));
>> 
>> First, and most importantly, C's "^" operator performs bitwise
>> exclusive-or (XOR), not exponentiation. Maybe you meant:
>> 
>> H_Array=(unsigned int *)G_calloc(1<<32, sizeof(unsigned int));
>> ?
>> 
>> But even that is wrong; if you have a 32-bit "int" type (and almost
>> every OS in common use does), 1<<32 == 0 according to the rules of C.
>> Even if size_t is a 64-bit type (which is quite likely on a 64-bit
>> system). The reason is that:
>> 
>> 1. C types propagate from the bottom up; an expressions comprised
>> solely of "int" values will be truncated to an "int", even if the
>> result is assigned to a variable or parameter of a wider type (e.g.
>> "long").
>> 
>> 2. Undecorated integer literals have type "int"; if you want a long
>> value, you need e.g. 1L.
>> 
>> The following should work if your platform's "size_t" is 64 bits:
>> 
>> H_Array=(unsigned int *)G_calloc(1L<<32, sizeof(unsigned int));
>> 
>> Finally:
>> 
>> 1. As Paul has pointed out, the number of entries and their types are
>> unrelated.
>> 
>> 2. If you are on a 32-bit platform, there is no way that you can have
>> an array containing 2^32 "int"s, or even 2^32 bytes; The process'
>> address space is limited to 2^32 bytes in total, and some of that is
>> already taken. Even if you have more than 4GiB or RAM and/or swap, you
>> can't have more than 4GiB for an individual process.
>> 
>> -- 
>> Glynn Clements <glynn at gclements.plus.com> 
>
>