IMAP Search skipping results

[Mark Waddingham]

On 2020-03-23 15:21, Pi Digital via use-livecode wrote:
> Thanks Mark, your input here is appreciated.
> 
> This reply in that forum wasn’t helpful, was it
>> 
>> > Known bug #90 was reported in 2014. However it still occurs in 2019. Does
>> > anybody know how to overcome this situation?
>> 
>> Yes: by fixing the code!
>> 
> In the thread it talks of pagination (as do you) but doesn’t give an
> example of how to do it. How would we implement this in LC? If it is
> not giving us the correct count how do we know how many pages we will
> have to allow for and so on.

That's where I got the reference to pagination from - as I said it's not 
a particularly useful post there. I suspect the person replying assumed 
the person asking the question had a detailed understanding of IMAP 
queries (my knowledge of interacting with IMAP servers is more than it 
was earlier on, but still not really sufficient to give any detailed 
advice).

The tsNet support for IMAP comes entirely from cURL - so it works pretty 
much the same as the 'curl' shell command.

I found this with a quick search which might help:

   <https://gist.github.com/akpoff/53ac391037ae2f2d376214eac4a23634>

The queries listed there at least appear to give ways to query the count 
of things in a mailbox.

This page also looks like it might be helpful:

   <https://nbsoftsolutions.com/blog/introduction-to-imap>

Also this has some more info on using CURL to talk to IMAP:

   
<https://debian-administration.org/article/726/Performing_IMAP_queries_via_curl>

Having pondered this a bit, I'm not sure that even if cURL guys fixed 
the above bug it would actually make a difference to what you need to do 
to have 100% correct code. The method of fetching all UIDs of messages 
in a mailbox at once is completely unscalable - imagine an INBOX with 
100,000s of messages for example over a slow/flaky connection; or an 
INBOX which has a lot of traffic and changes exceptionally frequently; 
or sits on a heavily loaded server (you can imagine that no server would 
like to be polled for complete message lists all the time by lots of 
clients).

The IMAP protocol appears to be designed to be treated as something to 
use to synchronize local and remote state. Indeed, there is quite an 
extensive description of how to 'synchronize with an IMAP server' in an 
RFC:

   <https://tools.ietf.org/html/rfc4549#page-18>

Indeed, it states in one place:

<quote>
   The following is an example of the first FETCH:

    C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE
        RFC822.SIZE)

    Note 1: The first FETCH may result in the server's sending a huge
    volume of data.  A smart disconnected client should use message
    ranges (see also Section 3.2.1.2 of [RFC2683]), so that the user is
    able to execute a different operation between fetching information
    for a group of new messages.
</quote>

The unique id used to perform the sync operation is the UID; you can 
fetch ranges of those as they are 'guaranteed' to always increase and 
never be re-used. (The only caveat is the UIDVALIDITY, which is 
basically a cache-generation number - i.e. if the UIDVALIDITY changes 
you have to dump your local cache and start from scratch as it means the 
meaning of individual UIDs has changed).

Not sure how much the above helps...

Warmest Regards,

Mark.

P.S. I've been chatting to Seb about the issue (he reported the bug 
originally) - he has confirmed that the problem affects Kognition's IMAP 
code too (that component is still in development at this stage which is 
why it hadn't been noticed until I asked him to check). I had hoped the 
approach there might have been slightly different and so provided a 
viable workaround, but it would appear that it is not the case :(

-- 
Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps