IMAP Search skipping results

Pi Digital sean at pidigital.co.uk
Mon Mar 23 13:14:07 EDT 2020


Thanks Brian and Mark

A combination may well help. 

I am not for this project needing ALL but have been using UNSEEN. The mailboxes we are referencing have around 100-1200 emails coming in from O2 every morning at 4am. These then get processed at 5am to update their databases. I will have to first find the FIRST UNSEEN UID and then run a process to get the UIDs of the next 100 (and repeat) till they run out I guess. Once I’ve built the list of UIDs I can carry on as normal processing each one in turn. 

Finding the first unseen UID though seems to be the next issue I’m going to have to overcome. A never ending stream of workarounds. 

Sean Cole
Pi Digital Productions Ltd


eMail Ts & Cs


> On 23 Mar 2020, at 16:48, Mark Waddingham via use-livecode <use-livecode at lists.runrev.com> wrote:
> 
> On 2020-03-23 15:21, Pi Digital via use-livecode wrote:
>> Thanks Mark, your input here is appreciated.
>> This reply in that forum wasn’t helpful, was it
>>> > Known bug #90 was reported in 2014. However it still occurs in 2019. Does
>>> > anybody know how to overcome this situation?
>>> Yes: by fixing the code!
>> In the thread it talks of pagination (as do you) but doesn’t give an
>> example of how to do it. How would we implement this in LC? If it is
>> not giving us the correct count how do we know how many pages we will
>> have to allow for and so on.
> 
> That's where I got the reference to pagination from - as I said it's not a particularly useful post there. I suspect the person replying assumed the person asking the question had a detailed understanding of IMAP queries (my knowledge of interacting with IMAP servers is more than it was earlier on, but still not really sufficient to give any detailed advice).
> 
> The tsNet support for IMAP comes entirely from cURL - so it works pretty much the same as the 'curl' shell command.
> 
> I found this with a quick search which might help:
> 
>  <https://gist.github.com/akpoff/53ac391037ae2f2d376214eac4a23634>
> 
> The queries listed there at least appear to give ways to query the count of things in a mailbox.
> 
> This page also looks like it might be helpful:
> 
>  <https://nbsoftsolutions.com/blog/introduction-to-imap>
> 
> Also this has some more info on using CURL to talk to IMAP:
> 
>  <https://debian-administration.org/article/726/Performing_IMAP_queries_via_curl>
> 
> Having pondered this a bit, I'm not sure that even if cURL guys fixed the above bug it would actually make a difference to what you need to do to have 100% correct code. The method of fetching all UIDs of messages in a mailbox at once is completely unscalable - imagine an INBOX with 100,000s of messages for example over a slow/flaky connection; or an INBOX which has a lot of traffic and changes exceptionally frequently; or sits on a heavily loaded server (you can imagine that no server would like to be polled for complete message lists all the time by lots of clients).
> 
> The IMAP protocol appears to be designed to be treated as something to use to synchronize local and remote state. Indeed, there is quite an extensive description of how to 'synchronize with an IMAP server' in an RFC:
> 
>  <https://tools.ietf.org/html/rfc4549#page-18>
> 
> Indeed, it states in one place:
> 
> <quote>
>  The following is an example of the first FETCH:
> 
>   C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE
>       RFC822.SIZE)
> 
>   Note 1: The first FETCH may result in the server's sending a huge
>   volume of data.  A smart disconnected client should use message
>   ranges (see also Section 3.2.1.2 of [RFC2683]), so that the user is
>   able to execute a different operation between fetching information
>   for a group of new messages.
> </quote>
> 
> The unique id used to perform the sync operation is the UID; you can fetch ranges of those as they are 'guaranteed' to always increase and never be re-used. (The only caveat is the UIDVALIDITY, which is basically a cache-generation number - i.e. if the UIDVALIDITY changes you have to dump your local cache and start from scratch as it means the meaning of individual UIDs has changed).
> 
> Not sure how much the above helps...
> 
> Warmest Regards,
> 
> Mark.
> 
> P.S. I've been chatting to Seb about the issue (he reported the bug originally) - he has confirmed that the problem affects Kognition's IMAP code too (that component is still in development at this stage which is why it hadn't been noticed until I asked him to check). I had hoped the approach there might have been slightly different and so provided a viable workaround, but it would appear that it is not the case :(
> 
> -- 
> Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
> LiveCode: Everyone can create apps
> 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode



More information about the use-livecode mailing list