[imapfilter-devel] Optimizing complicated searches

[William Faulk]

IMAPFilter doesn't seem to try to optimize its searching when performing 
functions that require it to fetch data from the server, rather than 
letting the server do the work.  Actually, I don't think it optimizes 
when it lets the server do the work, either, but it seems less relevant 
in that situation.

If I have a search like this:

r = connection.INBOX:match_subject('^Backup:')

IMAPFilter fetches the Subject for all messages in the inbox.  Fair 
enough, there's not really anything else it can do.

But if I have a search like this:

r = connection.INBOX:contain_from('backup-admin') *
     connection.INBOX:match_subject('^Backup:')

It still fetches every Subject in the inbox and it has the server search 
every From field.  It ought to be able to limit one of them to the 
results of the other.

That is, if the contain_from() returns only ten messages, then there's 
no reason for IMAPFilter to be fetching any more than ten Subjects.

I don't see a great way for IMAPFilter to know the best operation to do 
first.  Maybe just do them in order and let the user worry about which 
order is best optimized.  Maybe always try to optimize for lower data 
transfer, letting the server do most of the work, or have an option to 
choose between that or the opposite.

Or maybe it makes sense to force that onto the user directly by allowing 
IMAPFilter's functions to be called directly against a table of 
messages, like:

from_backup = connection.INBOX:contain_from('backup-admin')
r = from_backup:match_subject('^Backup:')

The impetus for this is the idea that IDLE support would have allowed me 
to run filters only against the message(s) that just arrived, speeding 
up the time for filters to take effect by quite a lot.  But it turned 
out to not work that way.

Obviously, I assume that I haven't missed the piece of documentation 
about how to perform this optimization as IMAPFilter already exists. 
Feel free to berate me if I have.

-Bitt