[imapfilter-devel] Re: Imapfilter sometimes doesn't filter

[David DeSimone]

David DeSimone <fox at verio.net> wrote:
>
> What would be really nice, I think, is if the Lua language had some
> sort of support for performing intersections or merges of lists.

I have since learned enough Lua to realize that this capability can be
invented within the language.


In case anyone is interested in this problem, let me repeat it here.

If you write some simple Lua code like this:

    results = match(acct, inbox, some_query)
    move(acct, inbox, acct, some_mbox, results)

    results = match(acct, inbox, {})
    move(acct, inbox, acct, outbox, results)

The intent of this code is that all messages matching "some_query" get
moved into "some_mbox", and all remaining messages get move to "outbox".

However, this code sometimes fails, due to race conditions.  If a
message happens to be delivered to INBOX between the first move and the
second, that message will end up in outbox, even if it would have
matched the previous filtering criteria.

I have found what I believe to be a solution to this problem.  At the
start of the code, generate a list of all messages, and then as messages
are moved to other folders, remove those id's from the master list. 
Then, only move the messages that remain in this list.

With this scheme, any new messages that arrive that are not filtered
into specific boxes will simply be left in INBOX until the next time
imapfilter runs.  I find this to be acceptable, since I run imapfilter
in daemon mode with a 45-second timer.

Here is the code that I wrote to make this work.  First, some support
functions are needed:

    function remove_from(list, item)

	if (list == nil or item == nil)
	then
	    return
	end

	local pos

	for k, v in pairs(list)
	do
	    if (v == item)
	    then
		pos = k
		break
	    end
	end

	if (pos ~= nil)
	then
	    table.remove(list, pos)
	end
    end

The above routine searches for an item in a list, and removes it if
found.

    function remove_list(master, remove)

	if ((master == nil) or (remove == nil))
	then
	    return
	end

	for k, v in pairs(remove)
	do
	    remove_from(master, v)
	end
    end

The routine above removes elements collected in one list, from another
list.  With these tools, it's now possible to write some code:

    exists, recent, unseen = check(account, "INBOX")

    if (exists == 0)
    then
	return
    end

This is an optimization, so that we don't waste time looking for
messages if there are none.

    all = match(account, "INBOX", {})

We get a list of all messages in the inbox here, and call it "all".

    query =
    {
	'smaller 8000',
	{
	    'header "List-Id" "unix-admin"',
	    'header "List-Id" "itops-maintenance"',
	    'header "List-Id" "it-serverops"'
	},
    }

    results = match(account, "INBOX", query)

    if (results ~= nil and table.getn(results) > 0)
    then
	move(account, "INBOX", account, "unix", results)
	remove_list(all, results)
    end

This is a basic query block.  As you can see, after moving the messages,
all message id's that were moved are removed from the "all" list.

    query =
    {
	'header "List-Id" "imapfilter-devel.lists.hellug.gr"'
    }

    results = match(account, "INBOX", query)

    if (results ~= nil and table.getn(results) > 0)
    then
	move(account, "INBOX", account, "imapfilter", results)
	remove_list(all, results)
    end

The basic query block repeats several times, for each folder that I want
to move messages to.

    if (all ~= nil and table.getn(all) > 0)
    then
	move(account, "INBOX", account, "Default", all)
    end

Finally, after all the queries have been run, we move any remaining
messages into the default folder.  My, that looks so simple now!  I
guess I am getting the hang of this Lua-nguage.  :)

-- 
David DeSimone || Network Admin || fox at verio.net
  "It took me fifteen years to discover that I had no
   talent for writing, but I couldn't give it up because
   by that time I was too famous.  -- Robert Benchley