file size issue

[Lowell Filak]

The following message was sent by Jerl Simpson <jerl.simpso at GMAIL.COM>
on Thu, 16 Jun 2005 08:41:02 -0500.

> Hello,
> 
> I have a particular issue with file size limits.  I have a layer (2
> actually) that contains roads classified as "Local Roads" throughout
> the entire contiguous 48 US states.  I had to split this up into 2
> files due to filesize limits.  My apache server keeps throwing errors
> that say "[notice] child pid 12855 exit signal File size limit
> exceeded (25)".  So I would like to add these to a MySQL table and
> pull the data from there.
> 
> I am running into two problems.
> 
> 1.  I used one of my very small shape files to setup an SQL connection
> to grab the data.  When I modified my .map file I now get an internal
> server error.  Which simply means some data was spit out before HTTP
> headers terminated.  I have a log file and DEBUG ON set in my map file
> and nothing gets placed in the file.  Is there any documentation that
> tells one how to build the queries?  I can't seem to find them.
> 
> 2.  The shape file I have for Local Roads is 1.2Gb.  And the perl
> Shapelib module reads the entire shape file into an array.  Well, I
> don't  have 1.2Gb of memory on my machine available so when I run
> "shp2mysql.pl LocalRoad" the program terminates with the error
> "Terminated".  I looked at the documentation and I don't see a way to
> just grab records out of the file.  I tried get_record() but that
> threw error messages.
> 
> I have a tiled this file into 100 smaller files the largest of which
> is 63Mb.  I could write a script to run to shp2mysql.pl on each file. 
> I would rather have a program to operate on the original file though,
> as I'm afraid I'll end up with 100 seperate tables with the same data.
>  Or I would need to modify the script to foce them all into the same
> table.

Jerl,

If you want to venture into modifying the shp2mysql.pl what can be done
is to write each record of data to a tab delimited text file and then
use 'load data local infile' instead of the insert.

HTH

Lowell