Making Revolution faster with really big arrays

Tue Apr 12 20:14:51 EDT 2005

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rev is still confined to what the Operating System offers it, which is 
2GB for most 32-bit systems, and never 4GB (3GB, or maybe even 3.5GB in 
some rare cases), as per my previous note.

I just noticed that Tiger "hits the shelves" in 17 days.  Sweet.

However, even in Tiger, 64-bit support is limited to "faceless" 
background apps (like database servers), and is *not* supported for 
developing with the Carbon and Cocoa frameworks.  Thus, Rev will not 
likely be releasing a 64-bit Mac version for some time, and the current 
version is certainly not a 64-bit app.  Rats.

On Apr 12, 2005, at 7:42 PM, Dennis Brown wrote:

>
> On Apr 12, 2005, at 7:37 PM, Dennis Brown wrote:
>
> Frank,
>
> I just copied this from the Rev IDE help info:
>
> A note about entries designated as "Unlimited":
> Since each open stack file resides completely in memory, Revolution 
> stacks (and all structures within a stack) are effectively limited by 
> available memory and by Revolution's total address space of 4G 
> (4,294,967,296 bytes) on 32-bit systems, or 16P 
> (18,446,744,073,709,551,616 bytes) on 64-bit systems.
>
> The 64KB limit on lines is also well within my data size.
>
> Anyway, with my scheme of keeping the data in 40MB files that I only 
> need several of in memory at a time, means memory is no longer my 
> limiting factor.  All I need is speed...
>
> If I can't do it in Rev, then I will try RB, or write portions in 
> each.  I have years of experience with X languages, years with 
> microprocessor machine code, and nothing in between.  So everything 
> else is a learning curve, and I have to have some results from this 
> project in a couple of weeks.  My gripe is that Rev is only lacking a 
> small enhancement to open up the possibility of quite high speed data 
> processing, it is a crying shame.
>
> Dennis
>
> On Apr 12, 2005, at 6:27 PM, Frank D. Engel, Jr. wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> That is only significant if Rev takes advantage of the 64-bit 
>>> address space, which I seriously doubt.  Your Rev process will still 
>>> be limited to 2GB of addressing space, regardless of how much RAM is 
>>> in the system.  Until they release a 64-bit version of Rev, of 
>>> course...
>>>
>>> If your task is that processor-intensive and your data set that 
>>> large, you should consider a lower-level language like Pascal or 
>>> Ada.  A scripting language, no matter how fast it is, is not ideal 
>>> for such intensive operations on large data sets.
>>>
>>>
>>> On Apr 12, 2005, at 5:30 PM, Dennis Brown wrote:
>>>
>>>> Thanks Frank,
>>>>
>>>> Actually I have a 3.5GB 64 bit G5 machine that can handle that much 
>>>> data, and I could add a couple more gig if I needed to.  It crashes 
>>>> when I get less than 1GB into RAM (I can monitor the number of free 
>>>> pages of RAM).  I tried processing it like you suggest.  However, 
>>>> at the speed it was going, it was going to be 4 or 5 days to get 
>>>> the first pass of my data processed.  That is because if you 
>>>> specify a line or item chunk in a big array, Rev counts separators 
>>>> from the beginning to find the spot you want each time, even if you 
>>>> just want the next line.  That means on the average, you have 
>>>> processed the array thousands of more times than the single pass 
>>>> repeat for each takes.  The way I wrote it, it only required about 
>>>> two hours for the initial pass, and about two minutes for single 
>>>> passes through one data item in the array.  However, now I need to 
>>>> process more than one data item at a time, and that means I can use 
>>>> the repeat for each on only one item and I will have to use the 
>>>> chunk expressions for the others.  That will slow me back down to 
>>>> many days per pass, and I have hundreds of passes to do --not very 
>>>> interactive!  See you in a few years...
>>>>
>>>> Dennis
>>>>
>>>>
>>>> On Apr 12, 2005, at 5:04 PM, Frank D. Engel, Jr. wrote:
>>>>
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA1
>>>>>
>>>>> Rev's arrays are associative.  When using an array with an index 
>>>>> like [X, Y, Z], you are really saying, make a string whose 
>>>>> contents are X, Y, and Z separated by commas, then use that as the 
>>>>> index for the array.  These array indexes take up memory, along 
>>>>> with your data.  In fact, depending on what type of data you are 
>>>>> trying to process, they likely take up more.  Even without the 
>>>>> overhead of the structures used to represent the arrays, your 
>>>>> array will likely take up well over 2GB of RAM.  On a 32-bit 
>>>>> system, you are normally limited to either 2GB or 3GB of memory 
>>>>> per process (almost always 2GB, but some Windows versions -- 
>>>>> mostly server versions -- can be configured for 3GB per process), 
>>>>> so that array would take more memory than all of your data PLUS 
>>>>> Revolution PLUS your stack(s) PLUS some code used by the runtime 
>>>>> libraries from the OS ... you get the idea.
>>>>>
>>>>> You'll never be able to fit that entire array into memory *as an 
>>>>> array* in Rev.
>>>>>
>>>>> Have you considered loading it into a single string and parsing 
>>>>> the data inline while managing it in your code?
>>>>>
>>>>> Try something like:
>>>>>
>>>>> put URL "file:/path/to/MyFile.txt" into x
>>>>>
>>>>> Then parse the data from x:
>>>>>
>>>>> put word 1 of item 2 of line 6 of x into y
>>>>>
>>>>> And so on...
>>>>>
>>>>>
> _______________________________________________
> use-revolution mailing list
> use-revolution at lists.runrev.com
> http://lists.runrev.com/mailman/listinfo/use-revolution
>
>
- -----------------------------------------------------------
Frank D. Engel, Jr.  <fde101 at fjrhome.net>

$ ln -s /usr/share/kjvbible /usr/manual
$ true | cat /usr/manual | grep "John 3:16"
John 3:16 For God so loved the world, that he gave his only begotten 
Son, that whosoever believeth in him should not perish, but have 
everlasting life.
$
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)

iD8DBQFCXGR77aqtWrR9cZoRAhF8AJ9DxDY3jrLqX1gJ3K1aB/mGMjMhgQCdHLcy
sfOFAxh6Vr23xg81sDrW5CI=
=eRxW
-----END PGP SIGNATURE-----

___________________________________________________________
$0 Web Hosting with up to 200MB web space, 1000 MB Transfer
10 Personalized POP and Web E-mail Accounts, and much more.
Signup at www.doteasy.com