[imapfilter-devel] Optimizing complicated searches

[Lefteris Chatzimparmpas]

Heh, I believe we were writing similar things at the same time just on a
different thread (ie. IMAPFilter 2.1) :-)

What you describe is almost the same as what I had in mind to implement next,
and I actually came up with the pipelining thing last night after discussing it
with a friend.

Imapfilter provided match_*() methods as a bonus when IMAP searching wasn't
enough, and then it was left without any more work on that front (or any front)
for quite some time.  Altough you could always search, then do fetch_*() on the
messages search returned and match locally on the downloaded parts, and put
that on a nice function.  Anyway, hopefully, I will be able to pay more
attention to it now.

As is the code now, it can be hacked in half an hour to add an extra argument
to all the match_*() methods; this argument will be the messages to download,
instead of doing select_all() which is done by default now.  And this doesn't
require any changes in the C code, just play with the mailbox.lua Lua code.

With a bit more work it could be done in a more elegant way and that's what
I've been thinking, on how to do it better and easier to be useful and
powerful...

Apart from that, I have some more cards up my sleeve on how to even use the
same filter on different mailboxes, eg.

	account1.INBOX:is_recent() + account2.INBOX:is_recent()

This would return messages from both mailboxes on different servers and then
whatever you do to them (eg. delete) would be done transparently by imapfilter,
because it would know which message belongs to which server/mailbox.

Interesting ideas that I plan to get my hands on when I have some free
time to see what can come out of them...


On Wed, Nov 25, 2009 at 04:35:29PM -0500, William Faulk wrote:
> IMAPFilter doesn't seem to try to optimize its searching when performing  
> functions that require it to fetch data from the server, rather than  
> letting the server do the work.  Actually, I don't think it optimizes  
> when it lets the server do the work, either, but it seems less relevant  
> in that situation.
>
> If I have a search like this:
>
> r = connection.INBOX:match_subject('^Backup:')
>
> IMAPFilter fetches the Subject for all messages in the inbox.  Fair  
> enough, there's not really anything else it can do.
>
> But if I have a search like this:
>
> r = connection.INBOX:contain_from('backup-admin') *
>     connection.INBOX:match_subject('^Backup:')
>
> It still fetches every Subject in the inbox and it has the server search  
> every From field.  It ought to be able to limit one of them to the  
> results of the other.
>
> That is, if the contain_from() returns only ten messages, then there's  
> no reason for IMAPFilter to be fetching any more than ten Subjects.
>
> I don't see a great way for IMAPFilter to know the best operation to do  
> first.  Maybe just do them in order and let the user worry about which  
> order is best optimized.  Maybe always try to optimize for lower data  
> transfer, letting the server do most of the work, or have an option to  
> choose between that or the opposite.
>
> Or maybe it makes sense to force that onto the user directly by allowing  
> IMAPFilter's functions to be called directly against a table of  
> messages, like:
>
> from_backup = connection.INBOX:contain_from('backup-admin')
> r = from_backup:match_subject('^Backup:')
>
> The impetus for this is the idea that IDLE support would have allowed me  
> to run filters only against the message(s) that just arrived, speeding  
> up the time for filters to take effect by quite a lot.  But it turned  
> out to not work that way.
>
> Obviously, I assume that I haven't missed the piece of documentation  
> about how to perform this optimization as IMAPFilter already exists.  
> Feel free to berate me if I have.
>
> -Bitt
> _______________________________________________
> Imapfilter-devel mailing list
> Imapfilter-devel at lists.hellug.gr
> http://lists.hellug.gr/mailman/listinfo/imapfilter-devel