Problem parsing data in Gigabyte size text files
capellan2000 at yahoo.com
Wed Jul 4 18:02:33 CDT 2007
Recently, i was extracting data
from a 8 gigabyte ANSI text file
(a XML customer database), but after
processing approximately 3.5 gigabyte
of data, Revolution quits itself and
Windows XP presents the familiar dialog
asking to notify the Developer of this
The log file that i saved, while using
the stack, shows that after reading character
3,758,096,384 (that is more than 3 thousand million
of characters) the stack could not read anymore
into the XML database and start repeating the
same last line of text that it reads.
Notice that i checked the processor and memory use
with Windows Task Manager and everything was normal.
The stack was using between a 30 to 70 % of processor
and memory use was between 45 MB and 125 MB.
The code used is similar to this:
repeat until tCounter = 8589934592 -- 8 Gigabites
read from file tData from char tCounter for 10000
-- reading 10,000 characters from database
-- these character are placed in the variable: it
put processDATA(it) into tProcessedData
write tProcessedData to tNewFile
put tCounter && last line of it & cr after URL tLOG
add 10000 to tCounter
I have repeated the test at least 3 times :((
and the results are almost the same, with a small
difference between the character where stack quits,
while reading this 8 Gigabyte size XML database.
I have checked for strange characters in that part of
the database, when i splitted the file in many parts,
but have not found any.
Every insight that you could provide to process
this database from start to end is more
than welcome. :)
Thanks in advance.
Visit my site:
Sucker-punch spam with award-winning protection.
Try the free Yahoo! Mail Beta.
More information about the use-livecode