file size issue

[Jerl Simpson]

Lowell,

That's a good idea.  I was thinking of writing each INSERT query to a
separate file.  Then running that after I made sure everthing was
present and accounted for.

Thanks for the tip.

Jerl

On 6/16/05, Lowell Filak <lfilak at medinaco.org> wrote:
> The following message was sent by Jerl Simpson <jerl.simpso at GMAIL.COM>
> on Thu, 16 Jun 2005 08:41:02 -0500.
> 
> > Hello,
> >
> > I have a particular issue with file size limits.  I have a layer (2
> > actually) that contains roads classified as "Local Roads" throughout
> > the entire contiguous 48 US states.  I had to split this up into 2
> > files due to filesize limits.  My apache server keeps throwing errors
> > that say "[notice] child pid 12855 exit signal File size limit
> > exceeded (25)".  So I would like to add these to a MySQL table and
> > pull the data from there.
> >
> > I am running into two problems.
> >
> > 1.  I used one of my very small shape files to setup an SQL connection
> > to grab the data.  When I modified my .map file I now get an internal
> > server error.  Which simply means some data was spit out before HTTP
> > headers terminated.  I have a log file and DEBUG ON set in my map file
> > and nothing gets placed in the file.  Is there any documentation that
> > tells one how to build the queries?  I can't seem to find them.
> >
> > 2.  The shape file I have for Local Roads is 1.2Gb.  And the perl
> > Shapelib module reads the entire shape file into an array.  Well, I
> > don't  have 1.2Gb of memory on my machine available so when I run
> > "shp2mysql.pl LocalRoad" the program terminates with the error
> > "Terminated".  I looked at the documentation and I don't see a way to
> > just grab records out of the file.  I tried get_record() but that
> > threw error messages.
> >
> > I have a tiled this file into 100 smaller files the largest of which
> > is 63Mb.  I could write a script to run to shp2mysql.pl on each file.
> > I would rather have a program to operate on the original file though,
> > as I'm afraid I'll end up with 100 seperate tables with the same data.
> >  Or I would need to modify the script to foce them all into the same
> > table.
> 
> Jerl,
> 
> If you want to venture into modifying the shp2mysql.pl what can be done
> is to write each record of data to a tab delimited text file and then
> use 'load data local infile' instead of the insert.
> 
> HTH
> 
> Lowell
>