I've been migrating a set of Mailman mailing lists to new
servers recently, and have implemented an extension to the
standard exim+Mailman configuration which may be of
interest to others. Comments welcome, of course.
This trick is suitable for lists where only members (and
perhaps a small set of known nonmembers or alternate
addresses for members) are permitted posters, and
prohibited messages are rejected outright rather than
being moderated later. (Many lists which start out with
another policy end up having this policy de facto, because
of the hassle of sorting a few legitimate posts from other
rubbish in Mailman's moderation queue.) Using a small
Python script to check the configuration of a destination
list, it is possible to refuse mail from non-permitted
senders before it even reaches Mailman.
Unfortunately (and as with doing anti-spam testing at SMTP
time) it is necessary to restrict each message transaction
to exactly one list recipient. This is because permission
to post to a mailing list is based on the From: header
(and perhaps also Reply-To:), rather than the envelope
sender. these values are known only after the
DATA command.
The necessary script is here:
http://tea.ukcod.org.uk/~
-- it calls into Mailman's own code, and may have to be
adapted for your local installation of Mailman (we use the
Debian packages which make an ambitious if ultimately
futile effort to contort Mailman's own arrangement of
files into something resembling the FHS). Because Mailman
uses on-disk files rather than a proper database, the
script needs to be able to read those files; in principle
one could do this with group membership, but (a) this made
me feel slightly uneasy security-wise; and (b) exim drops
supplementary group memberships so you'd have to have
Mailman run in exim's group rather than vice versa.
Therefore I instead implemented a restricted setuid
wrapper program, the source for which is here:
-- you could instead achieve the same effect with sudo or
userv or whatever, if that's what floats your boat. That
wrapper also permits the execution of another script,
http://tea.ukcod.org.uk/~
which, given the local-part of a list, returns its
corresponding domain name, which is useful if you have
Mailman lists in several different domains and don't want
them to clash with other sorts of names.
The relevant ACL configuration looks something like this:
(this is excerpted from our real mail config so is
probably not minimal)
- Firstly, set up some macros and a domain list (some of
these are in the standard howto):
domainlist mailman_domains =
MM_LISTCHECK = /var/lib/mailman/lists/${lc:$local_part}/config.pck
MM_QUERY = /
# these are arbitrary
MAILMAN_LIST = acl_m1
MAILMAN_DMAIN = acl_m2
- Then, in the RCPT ACL, enforce the `one list
delivery per transaction' constraint, and record some
information about the list for later:
# Special case for list mail. We want to ensure that each message is
# submitted to either exactly one mailing list, or any number of
# non-mailing-list addresses per submission, so that we can check that the
# sender is permitted at the end of the DATA statement.
defer domains = +mailman_domains
condition = ${if exists{MM_LISTCHECK} \
{${if >{${eval:$rcpt_count \
- $rcpt_defer_count \
- $rcpt_fail_count}}{1} \
{yes} {no} }} \
{no} }
message = Please make each Mailman delivery in a separate submission
# Record information about a list delivery
warn domains = +mailman_domains
condition = ${if exists{MM_LISTCHECK} }
set MAILMAN_LIST = ${lc:$local_part}
set MAILMAN_DMAIN = ${lc:$domain}
message = destination is Mailman list ${lc:$local_part}
- Finally, in the DATA ACL, call out to the script to
test whether a given list delivery is permitted:
# If this message is to a Mailman list, check that the sender is permitted to
# post to the list. This is slightly nasty, because Mailman (correctly) uses
# the From: header to decide whether the sender is authorised, which is why
# we need to do this test in the DATA ACL rather than at recipient time.
deny message = You are not permitted to send mail to the $MAILMAN_LIST \
mailing list. Please contact \
$MAILMAN_LIST-owner@$MAILMAN_DMAIN for further \
information.
# Mailman considers the "sender" of a mail to be the From: address,
# envelope sender, Reply-To: address or Sender: address. For our
# purposes the order does not matter.
condition = ${if !eq{$MAILMAN_LIST}{} \
{${run {MM_QUERY test-sender $MAILMAN_LIST \
$sender_address \
${address:$h_From:} \
${address:$h_Reply-To:} \
} {no} {yes} }} \
{no} }
A couple of issues with this: firstly, the test-sender
script doesn't implement Mailman's full logic for
accepting/denying/queueing an incoming mail. It could be
extended to do so, though reimplementing it in the way I
do about is a bit ugly. Secondly, forking and execing the
script is going to be slow, but in most installations this
probably won't matter (if it does then write a daemon
which responds to queries on some socket; that would also
get around the irritating privilege-boundary issue).
More generally this has got me wondering about how much of
a mailing-list manager could be implemented within a
modern MTA. It is painfully obvious that most of Mailman's
queueing and aliasing stuff could equally well be
implemented with Exim and a database and suitable
transports. More difficult is what to do with the
moderation queue aspect of the problem, which is required
in some mailing lists.
evil possibility would be to defer mail in an ACL,
while recording a copy in the moderation queue; the result
of moderation could then be used to decide whether to
accept or decline the message the next time the sending
MTA submits it (identifying the message by its Message-ID:
or perhaps a loose hash of its contents). That has some
bad properties (the message might never be resubmitted;
there's a delay between moderator approval and the message
reaching the list). Alternatively on approval the mail
could be passed to the list, and the next time the sending
MTA resubmits it, it could be accepted and silently
dropped, having already reached its recipients; of course,
that has the converse bad property that if it's never
resubmitted the sender will have received a misleading
error report. Perhaps this is just one of those things
which can't quite be sensibly done at SMTP time.

Mon, 3 Jul 2006, Chris Lightfoot wrote:

More generally this has got me wondering about how much of a
mailing-list manager could be implemented within a modern MTA. It is
painfully obvious that most of Mailman's queueing and aliasing stuff
could equally well be implemented with Exim and a database and suitable
transports. More difficult is what to do with the moderation queue
aspect of the problem, which is required in some mailing lists.

Yes, the coupling between transport, database, and user interface is one
of the nastier aspects of Mailman's architecture.

Exim isn't quite studly enough to do all the things that Mailman offers,
in particular attachment policies and general MIME mangling. If Exim were
to have full support for SMTP extensions like 8BITMIME, BINARYMIME, and
UTF8SMTP, it would have to have a MIME parsing and mangling
infrastructure, and so fancy MLM stuff would be less of a problem to
implement.

For the moderation queue, probably the best way would be to deliver the
messages into an IMAP mailbox, which could be accessed by the MLM front
end using the usual webmail techniques. The MLM could even re-submit
moderated messages for transport using BURL (RFC 4468). Then your user
interface, transport, storage, and database infrastructure can be cleanly
separated using open protocols.

Speculation aside, your early checking stuff looks nice, especially since
we are replacing our old and busted list system with Mailman's new hotness

Tony.

Mon, Jul 03, 2006 at 05:19:00PM +0100, Tony Finch wrote:
Mon, 3 Jul 2006, Chris Lightfoot wrote:

More generally this has got me wondering about how much of a
mailing-list manager could be implemented within a modern MTA. It is
painfully obvious that most of Mailman's queueing and aliasing stuff
could equally well be implemented with Exim and a database and suitable
transports. More difficult is what to do with the moderation queue
aspect of the problem, which is required in some mailing lists.

Yes, the coupling between transport, database, and user interface is one
of the nastier aspects of Mailman's architecture.

Exim isn't quite studly enough to do all the things that Mailman offers,
in particular attachment policies and general MIME mangling. If Exim were
to have full support for SMTP extensions like 8BITMIME, BINARYMIME, and
UTF8SMTP, it would have to have a MIME parsing and mangling
infrastructure, and so fancy MLM stuff would be less of a problem to
implement.

ok. This sort of functionality could be done at present,
by tagging messages with information about the required
processing steps at the ACL stage (either in the acl_m*
variables or in a header added at SMTP time) and then
processing them in a transport filter (or a local scan
function, though I see that modifying the message at this
stage is discouraged). That's not very nice, but then none
of this plumbing ever is, really

that said, doing all of the necessary processing in an
ACL could become a little bit unwieldy. It would be useful
to be able to call some arbitrary external code from an
ACL without the fork-and-exec cost and with something
slightly more convenient than the UNIX socket read/write
interface. How about a general `pass message and envelope
to some external process over [authenticated] HTTP, get
response status and content' string expansion function?
That's a bit nasty, I admit, but (I argue) no more so than
the package-specific protocols for ClamAV, SpamAssassin,
etc

For the moderation queue, probably the best way would be to deliver the
messages into an IMAP mailbox, which could be accessed by the MLM front
end using the usual webmail techniques. The MLM could even re-submit
moderated messages for transport using BURL (RFC 4468). Then your user
interface, transport, storage, and database infrastructure can be cleanly
separated using open protocols.

noted -- I hadn't started to think about the mechanics of
moderation yet.

I remain (idly) interested in whether it is possible to
exploit SMTP-time error-reporting to do moderation without
ever having to send a `your message has been held' or a
`your message has been rejected' bounce back to the poster
/ spammer. I suspect the solution I outlined is too ugly
to be very popular, though.

Speculation aside, your early checking stuff looks nice, especially since
we are replacing our old and busted list system with Mailman's new hotness

yeah. If you need all of Mailman's functionality for
determining which posters are/aren't permitted, then you
should probably throw out my code and use something which
constructs a fake message object and passes it through as
many of the Mailman handlers as are relevant -- the
alternative is to cut and paste all of the rest of the ACL
logic in Mailman into the test script, which is not
sensible long-term. But for the restricted case I'm
interested in the current solution is working well :-)

3 July 2006 17:19:00 +0100 Tony Finch <dot (AT) dotat (DOT) atwrote:

Mon, 3 Jul 2006, Chris Lightfoot wrote:
>>
>More generally this has got me wondering about how much of a
>mailing-list manager could be implemented within a modern MTA. It is
>painfully obvious that most of Mailman's queueing and aliasing stuff
>could equally well be implemented with Exim and a database and suitable
>transports. More difficult is what to do with the moderation queue
>aspect of the problem, which is required in some mailing lists.
>
Yes, the coupling between transport, database, and user interface is one
of the nastier aspects of Mailman's architecture.

Exim isn't quite studly enough to do all the things that Mailman offers,
in particular attachment policies and general MIME mangling. If Exim were
to have full support for SMTP extensions like 8BITMIME, BINARYMIME, and
UTF8SMTP, it would have to have a MIME parsing and mangling
infrastructure, and so fancy MLM stuff would be less of a problem to
implement.

Well, that would be ideal, but actually probably isn't necessary if we're
just trying to solve the collateral spam problem. If a message is from a
person permitted to send to the list, then it *probably* isn't spam. That
means we don't mind sending them a bounce message. In fact, a bounce
message is likely to me more useful to the sender.

exceptions are where a well known address like postmaster@ is
allowed to post to the list.

Chris Lightfoot said:
[]
Unfortunately (and as with doing anti-spam testing at SMTP
time) it is necessary to restrict each message transaction
to exactly one list recipient. This is because permission
to post to a mailing list is based on the From: header
(and perhaps also Reply-To:), rather than the envelope
sender. these values are known only after the
DATA command.
I have similar problems using other MLM-software and I'd like to add my
thoughts
Currently I'm replacing listserv with symmpa. Mailman is of no interest
to me, because it is lacking too many features, e.g. active development
(for two years little more than bugfixes and an exiting roadmap for
mailman3 which hasn't been touched for ages).

Listserv ist evil, sending thousands of bounces like "you are not a
member of the list", "mail has been forwarded to moderator", "mail has
been rejected by moderator", etc.
I'd like to reduce those bounces, so when incoming e-mail is processed
by exim, it checks whether the mailinglist would probably accept this
message and denies after data if it can be certain that the message
would be rejected. The setup is rather similar (but inferior) to the one
deveoped by Chris.
The moderation-messages should not be sent, but sadly Listserv has no
way to say "if SpamAssassin says it's probably spam, sent it to the
moderator quietly" and the moderator can not even suppress the message
"your mail has been rejected by the moderator".

Moving to sympa some things do change
Sympa has a very sophisticated authentication-language to define how to
handle a message. The only reasonable way is not to check the
configuration yourself to decide how to handle a message, but to query
sympa and ask for a judgment.

Luckily it is very easy to use the sympa-perl-code, do some copy'n'paste
and create a Milter-daemon which can answer precisely how sympa will
handle the message. As has already been noted, such an attempt would be
a maintenance-nightmare, but I will to this at a
"proof-of-concept"-level, when I start to test exim's milter-support,
which Hilko Bengen is currently working at.

Given this milter and sympas capabilities, I expect to be able to reject
some messages during the SMTP-transaction and to suppress some bounces
like "forwarded to moderator" for Spam-messages having a high
SpamAssassin-Score.

Do you consider this to be worth the effort? Not just implementation and
maintenance of the code, but I'm worried about load as well. Currently
incoming mail is just dropped by MLM to an incoming queue and processed
when recourses are available, which wouldn't work with a
milter-interface anymore.

I really really want to reduce my bounces, not only because Spamcop
sometimes lists my mailinglist-server. But with mailinglists it is
awfully difficult. Any other ideas what could be done with reasonable
effort?

Tue, Jul 04, 2006 at 01:54:49PM +0200, Patrick von der Hagen wrote:
[]
Do you consider this to be worth the effort? Not just implementation and
maintenance of the code, but I'm worried about load as well. Currently
incoming mail is just dropped by MLM to an incoming queue and processed
when recourses are available, which wouldn't work with a
milter-interface anymore.

My experience so far is that this is working well, with
the one problem that the bounce messages generated by
remote MTAs are not as friendly as those which Mailman
itself generates. So some users who (it turns out) were
posting from various different addresses, some
non-subscribed, and then having posts approved manually
are losing out if they do not read the bounce message
carefully. (In practice almost no users read bounce
messages carefully, sadly.)

Load: depends on what your mail servers are struggling
with, but assuming that you have enough RAM to hold the
mailing list configs and disk I is the problem, then
doing the rejection at SMTP time should reduce load, since
Exim only has to write the queue files, and not actually
do a delivery off to some other program which must write
and sync the data to disk. (Has exim actually called sync
on those files at the time the DATA ACL is run?)

course, if CPU is the scarce resource (much less likely
IME), then this isn't true, and doing the sender
authentication twice will make things worse.

Tue, 4 Jul 2006, Patrick von der Hagen wrote:

Moving to sympa some things do change
Sympa has a very sophisticated authentication-language to define how to
handle a message. The only reasonable way is not to check the
configuration yourself to decide how to handle a message, but to query
sympa and ask for a judgment.

Luckily it is very easy to use the sympa-perl-code, do some copy'n'paste
and create a Milter-daemon which can answer precisely how sympa will
handle the message. As has already been noted, such an attempt would be
a maintenance-nightmare, but I will to this at a
"proof-of-concept"-level, when I start to test exim's milter-support,
which Hilko Bengen is currently working at.

Why not use ${perl?

Tony.

Tue, 4 Jul 2006, Chris Lightfoot wrote:
Has exim actually called sync on [the queue files] at the time the DATA
ACL is run?
Yes.
Tony.

Tue, Jul 04, 2006 at 02:18:20PM +0100, Tony Finch wrote:
Tue, 4 Jul 2006, Chris Lightfoot wrote:

Has exim actually called sync on [the queue files] at the time the DATA
ACL is run?

Yes.

hmm -- what's the reason for this? Surely you don't need
to call sync until immediately before issuing a 2xx
response to the client?

Tue, 4 Jul 2006, Chris Lightfoot wrote:

that said, doing all of the necessary processing in an ACL could
become a little bit unwieldy. It would be useful to be able to call some
arbitrary external code from an ACL without the fork-and-exec cost and
with something slightly more convenient than the UNIX socket read/write
interface. How about a general `pass message and envelope to some
external process over [authenticated] HTTP, get response status and
content' string expansion function? That's a bit nasty, I admit, but (I
argue) no more so than the package-specific protocols for ClamAV,
SpamAssassin, etc

It's really hard to do this in a way which isn't totally nasty.

idea is to have something like a whole-message SMTP call-forward,
which is effectively how Postfix's before-queue filtering works.

The disadvantages are that it limits the SMTP extensions you can use to
those supported by the filter, and it makes it harder to trace a message
through your logs.

Your HTTP call-aside idea is somewhat reminiscent of the PES architecture
(see ) but the WG seems
to be moribund.

Tony.

Tony Finch schrieb:
[]
Why not use ${perl?
Two reasons. My incoming servers have to check the message, but they
don't run sympa. So they have to push that query to the sympa-server,
have it perform its checks and get a result. Using a milter-interface, I
get this networking-stuff for free.
And of course, if I can solve a problem not only for exim but for other
MTAs as well, I'll prefer the generic solution.

4 July 2006 13:54:49 +0200 Patrick von der Hagen <patrick (AT) wudika (DOT) de
wrote:

Chris Lightfoot said:
[]
>Unfortunately (and as with doing anti-spam testing at SMTP
>time) it is necessary to restrict each message transaction
>to exactly one list recipient. This is because permission
>to post to a mailing list is based on the From: header
>(and perhaps also Reply-To:), rather than the envelope
>sender. these values are known only after the
>DATA command.
I have similar problems using other MLM-software and I'd like to add my
thoughts
Currently I'm replacing listserv with symmpa. Mailman is of no interest
to me, because it is lacking too many features, e.g. active development
(for two years little more than bugfixes and an exiting roadmap for
mailman3 which hasn't been touched for ages).

"exiting" or "exciting"?

Ian Eiloart schrieb:
[]
"exiting" or "exciting"?
Both. ;-)
It is existing and when it was created one could have considered it to
be exiting. I've just been to the new Mailman-Wiki, someone seems to be
working at Mailman 2.2 and put up a new roadmap
Personally, I'm very happy about that, but not even mailman 2.2 will
have the features I do require at my current employer. But given enough
time they might catch up with my current favorite. ;-)

Mon, 2006-07-03 at 16:47 +0100, Chris Lightfoot wrote:
evil possibility would be to defer mail in an ACL,
while recording a copy in the moderation queue; the result
of moderation could then be used to decide whether to
accept or decline the message the next time the sending
MTA submits it (identifying the message by its Message-ID:
or perhaps a loose hash of its contents).

I do something similar with fakereject and control=freeze/no_tell.

We pretend to reject the offending message, sending an appropriate
message back to the sender. The message itself is frozen on the queue
and can be 'moderated' as appropriate.

In fact I use this on someone else's system for messages which my own
servers would just reject out of hand -- the intention was that the
rejection message would point to a URL containing a 'captcha' which a
real person could use to unfreeze their genuine false positive.

But I keep meaning to do something based on this for replacing mailman,
which I grow to hate more and more every day :)