Problem parsing data in Gigabyte size text files

Dave dave at looktowindward.com
Thu Jul 5 04:33:00 EDT 2007


Hi,

There is a problem at least on the Mac Platform whereby RunRev cannot  
access files that are > 2GB, maybe this is a similar problem with  
WIndows?

I got around the problem by writing a few External Functions that  
call the OS (on Mac) which gets around the problem nicely (it's also  
a LOT faster!).

Hope this Helps
Dave

On 5 Jul 2007, at 00:02, Alejandro Tejada wrote:

> Hi all,
>
> Recently, i was extracting data
> from a 8 gigabyte ANSI text file
> (a XML customer database), but after
> processing approximately 3.5 gigabyte
> of data, Revolution quits itself and
> Windows XP presents the familiar dialog
> asking to notify the Developer of this
> error.
>
> The log file that i saved, while using
> the stack, shows that after reading character
> 3,758,096,384 (that is more than 3 thousand million
> of characters) the stack could not read anymore
> into the XML database and start repeating the
> same last line of text that it reads.
>
> Notice that i checked the processor and memory use
> with Windows Task Manager and everything was normal.
> The stack was using between a 30 to 70 % of processor
> and memory use was between 45 MB and 125 MB.
>
> The code used is similar to this:
>
> repeat until tCounter = 8589934592 -- 8 Gigabites
> read from file tData from char tCounter for 10000
> -- reading 10,000 characters from database
> -- these character are placed in the variable: it
> put processDATA(it) into tProcessedData
> write tProcessedData to tNewFile
> put tCounter && last line of it & cr after URL tLOG
> add 10000 to tCounter
> end repeat
>
> etc...
>
> I have repeated the test at least 3 times :((
> and the results are almost the same, with a small
> difference between the character where stack quits,
> while reading this 8 Gigabyte size XML database.
>
> I have checked for strange characters in that part of
> the database, when i splitted the file in many parts,
> but have not found any.
>
> Every insight that you could provide to process
> this database from start to end is more
> than welcome. :)
>
> Thanks in advance.
>
> alejandro
>
>
> Visit my site:
> http://www.geocities.com/capellan2000/
>
>
>
> ______________________________________________________________________ 
> ______________
> Sucker-punch spam with award-winning protection.
> Try the free Yahoo! Mail Beta.
> http://advision.webevents.yahoo.com/mailbeta/features_spam.html
> _______________________________________________
> use-revolution mailing list
> use-revolution at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution




More information about the Use-livecode mailing list