Problem parsing data in Gigabyte size text files

[Alejandro Tejada]

Hi all,

Recently, i was extracting data
from a 8 gigabyte ANSI text file
(a XML customer database), but after
processing approximately 3.5 gigabyte
of data, Revolution quits itself and
Windows XP presents the familiar dialog
asking to notify the Developer of this
error.

The log file that i saved, while using
the stack, shows that after reading character
3,758,096,384 (that is more than 3 thousand million 
of characters) the stack could not read anymore
into the XML database and start repeating the
same last line of text that it reads.

Notice that i checked the processor and memory use 
with Windows Task Manager and everything was normal.
The stack was using between a 30 to 70 % of processor
and memory use was between 45 MB and 125 MB.

The code used is similar to this:

repeat until tCounter = 8589934592 -- 8 Gigabites
read from file tData from char tCounter for 10000
-- reading 10,000 characters from database
-- these character are placed in the variable: it
put processDATA(it) into tProcessedData
write tProcessedData to tNewFile
put tCounter && last line of it & cr after URL tLOG
add 10000 to tCounter
end repeat

etc...

I have repeated the test at least 3 times :((
and the results are almost the same, with a small
difference between the character where stack quits,
while reading this 8 Gigabyte size XML database. 

I have checked for strange characters in that part of
the database, when i splitted the file in many parts,
but have not found any.

Every insight that you could provide to process
this database from start to end is more 
than welcome. :)

Thanks in advance.

alejandro


Visit my site:
http://www.geocities.com/capellan2000/


 
____________________________________________________________________________________
Sucker-punch spam with award-winning protection. 
Try the free Yahoo! Mail Beta.
http://advision.webevents.yahoo.com/mailbeta/features_spam.html