Problem parsing data in Gigabyte size text files
dave at looktowindward.com
Thu Jul 5 04:33:00 EDT 2007
There is a problem at least on the Mac Platform whereby RunRev cannot
access files that are > 2GB, maybe this is a similar problem with
I got around the problem by writing a few External Functions that
call the OS (on Mac) which gets around the problem nicely (it's also
a LOT faster!).
Hope this Helps
On 5 Jul 2007, at 00:02, Alejandro Tejada wrote:
> Hi all,
> Recently, i was extracting data
> from a 8 gigabyte ANSI text file
> (a XML customer database), but after
> processing approximately 3.5 gigabyte
> of data, Revolution quits itself and
> Windows XP presents the familiar dialog
> asking to notify the Developer of this
> The log file that i saved, while using
> the stack, shows that after reading character
> 3,758,096,384 (that is more than 3 thousand million
> of characters) the stack could not read anymore
> into the XML database and start repeating the
> same last line of text that it reads.
> Notice that i checked the processor and memory use
> with Windows Task Manager and everything was normal.
> The stack was using between a 30 to 70 % of processor
> and memory use was between 45 MB and 125 MB.
> The code used is similar to this:
> repeat until tCounter = 8589934592 -- 8 Gigabites
> read from file tData from char tCounter for 10000
> -- reading 10,000 characters from database
> -- these character are placed in the variable: it
> put processDATA(it) into tProcessedData
> write tProcessedData to tNewFile
> put tCounter && last line of it & cr after URL tLOG
> add 10000 to tCounter
> end repeat
> I have repeated the test at least 3 times :((
> and the results are almost the same, with a small
> difference between the character where stack quits,
> while reading this 8 Gigabyte size XML database.
> I have checked for strange characters in that part of
> the database, when i splitted the file in many parts,
> but have not found any.
> Every insight that you could provide to process
> this database from start to end is more
> than welcome. :)
> Thanks in advance.
> Visit my site:
> Sucker-punch spam with award-winning protection.
> Try the free Yahoo! Mail Beta.
> use-revolution mailing list
> use-revolution at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
More information about the Use-livecode