NULL characters and sorting

[Paul Looney]

Richard,
Thanks for the insight.
Most of this archived data had been imported from the previous  
HyperCard system. We've now done many such imports and this is the  
only customer with this problem.
Another customer, at the other end of the country, had a different  
problem; blank spaces in fields. For example, the description field  
might have 25 lines; lines 1 to 4 were correct, lines 5 to 9 were  
missing, and lines 10 to 25 were correct. The "hole" was not always  
in the same fields or the same lines. We traced this back to them  
using AppleWorks and MS Word to write notes - which they had then  
pasted into the notes field on the HC-based program.
We occasionally get a NULL in the current, Rev-based, system (about  
one in six months). We have checked the code but not found a source.  
Obviously the user can not type a NULL.
I will vote for both 7823 and, especially 7824.

On Mar 20, 2009, at 8:26 AM, Richard Gaskin wrote:

> Paul Looney wrote:
>
>> One of my customers had a problem displaying all of their  
>> archived  orders.
>> There should have been 16,020 archived records but only 3,879  
>> were  showing up in the list.
> ...
>> I started checking the contents of variables in the appropriate   
>> handler and discovered there were the proper number of records  
>> just  before a sort by column. After the sort, records were missing.
>> Phil, the Great, Davis - Wizard of West Linn - suggested checking  
>> for  and removing NULLs (because they terminate a line in C). They  
>> were  131,023 NULLs in the pre-sort variable.
>> When I removed them before the sort, the number of listed records   
>> jumped from 3,879 to 16,020.
>> This leaves some questions:
>> How can 131,023 NULL "characters" reduce the displayed "lines" by   
>> 12,141?
>
> The sort command has a limit which I don't believe is currently  
> documented:  it can only be used reliably on data sets in which no  
> line is longer than 65,535 characters.
>
> I've submitted a request to have this noted in the docs:
> <http://quality.runrev.com/qacenter/show_bug.cgi?id=7823>
>
> I've also submitted a request to have this limit raised:
> <http://quality.runrev.com/qacenter/show_bug.cgi?id=7824>
>
> I learned about this when I encountered a similarly mystifying bug  
> with one of my WebMerge customers, back when Dr. Raney managed the  
> engine. His response was that if one had an item in a line which  
> was pushing the line beyond those limits, that's a good argument  
> for putting that data somewhere else. ;)
>
> At the time I argued with him, but working around it led me to a  
> useful feature in my product: the ability to reference external  
> files from within a record.  In my case one of the editors at a  
> major Mac magazine was using WebMerge on some data from FileMaker,  
> in which he stored full articles.  With the addition of the new  
> feature he was able to write those articles in any tool and store  
> them anywhere, merely referencing the file from his database; my  
> product would then find it and include it as though it was part of  
> the data.
>
> While it worked out well for me and my customers, I still see the  
> occasional data set in which some lines are longer than 65,535  
> chars, and still believe it would be a useful enhancement to the  
> engine.
>
> In your case it may not be a problem at all now that the NULLs are  
> removed, presumably reducing the number of chars well below that  
> limit. 64k is approximately 28 pages' worth of stuff, so it's  
> pretty rare that such a limit would be exceeded on a single line of  
> actual data.
>
>
> Now that Phil's removed the NULLs, the next trick is to figure out  
> how those got into the data in the first place.  I'm tempted to  
> place a bet that it's related to pasting data that had been copied  
> from MS Word. I'd be interested to learn how that happened either way.
>
> --
>  Richard Gaskin
>  Fourth World
>  Revolution training and consulting: http://www.fourthworld.com
>  Webzine for Rev developers: http://www.revjournal.com
> _______________________________________________
> use-revolution mailing list
> use-revolution at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution