IMAP Search skipping results
mark at livecode.com
Mon Mar 23 12:48:13 EDT 2020
On 2020-03-23 15:21, Pi Digital via use-livecode wrote:
> Thanks Mark, your input here is appreciated.
> This reply in that forum wasn’t helpful, was it
>> > Known bug #90 was reported in 2014. However it still occurs in 2019. Does
>> > anybody know how to overcome this situation?
>> Yes: by fixing the code!
> In the thread it talks of pagination (as do you) but doesn’t give an
> example of how to do it. How would we implement this in LC? If it is
> not giving us the correct count how do we know how many pages we will
> have to allow for and so on.
That's where I got the reference to pagination from - as I said it's not
a particularly useful post there. I suspect the person replying assumed
the person asking the question had a detailed understanding of IMAP
queries (my knowledge of interacting with IMAP servers is more than it
was earlier on, but still not really sufficient to give any detailed
The tsNet support for IMAP comes entirely from cURL - so it works pretty
much the same as the 'curl' shell command.
I found this with a quick search which might help:
The queries listed there at least appear to give ways to query the count
of things in a mailbox.
This page also looks like it might be helpful:
Also this has some more info on using CURL to talk to IMAP:
Having pondered this a bit, I'm not sure that even if cURL guys fixed
the above bug it would actually make a difference to what you need to do
to have 100% correct code. The method of fetching all UIDs of messages
in a mailbox at once is completely unscalable - imagine an INBOX with
100,000s of messages for example over a slow/flaky connection; or an
INBOX which has a lot of traffic and changes exceptionally frequently;
or sits on a heavily loaded server (you can imagine that no server would
like to be polled for complete message lists all the time by lots of
The IMAP protocol appears to be designed to be treated as something to
use to synchronize local and remote state. Indeed, there is quite an
extensive description of how to 'synchronize with an IMAP server' in an
Indeed, it states in one place:
The following is an example of the first FETCH:
C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE
Note 1: The first FETCH may result in the server's sending a huge
volume of data. A smart disconnected client should use message
ranges (see also Section 184.108.40.206 of [RFC2683]), so that the user is
able to execute a different operation between fetching information
for a group of new messages.
The unique id used to perform the sync operation is the UID; you can
fetch ranges of those as they are 'guaranteed' to always increase and
never be re-used. (The only caveat is the UIDVALIDITY, which is
basically a cache-generation number - i.e. if the UIDVALIDITY changes
you have to dump your local cache and start from scratch as it means the
meaning of individual UIDs has changed).
Not sure how much the above helps...
P.S. I've been chatting to Seb about the issue (he reported the bug
originally) - he has confirmed that the problem affects Kognition's IMAP
code too (that component is still in development at this stage which is
why it hadn't been noticed until I asked him to check). I had hoped the
approach there might have been slightly different and so provided a
viable workaround, but it would appear that it is not the case :(
Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps
More information about the use-livecode