At 09:03 AM 10/21/2005, you wrote:
>Back in 2.64 There was the option to have SA stop going though rules
>then it hit a max score.
>Is there any option for this in 3.X.X ?

Nope.

All depends on the order of the rules. What if it had yet to come
across a whitelist rule?

From: "Chris L. Franklin" <cfranklin (AT) nomadcf (DOT) com>

Back in 2.64 There was the option to have SA stop going though rules
then it hit a max score.
Is there any option for this in 3.X.X ?

No. Such a rule actually costs processing time AND would create false
positives or false negatives entirely too easily.
{^_^}

jdow wrote:

From: "Chris L. Franklin" <cfranklin (AT) nomadcf (DOT) com>
>
>Back in 2.64 There was the option to have SA stop going though rules
>then it hit a max score.
>Is there any option for this in 3.X.X ?
>
>
No. Such a rule actually costs processing time AND would create false
positives or false negatives entirely too easily.
{^_^}

If you grouped the rulesets into +/- sets and processed all the - rules
first it wouldn't create false ly scored messages. would it ?

At 11:45 AM 10/21/2005, you wrote:
>If you grouped the rulesets into +/- sets and processed all the -
>rules first it wouldn't create false ly scored messages. would it ?

No, but you still would have to process all the rules regardless - I
mean a incorrectly whitelisted message could have enough spam content
to push it to a positive score, right?

And if you're system is that bogged down that you REALLY need to stop
scoring at say 6.0 - it's time to upgrade systems. :)

Evan Platt wrote:

At 11:45 AM 10/21/2005, you wrote:
>
>If you grouped the rulesets into +/- sets and processed all the -
>rules first it wouldn't create false ly scored messages. would it ?
>
>
No, but you still would have to process all the rules regardless - I
mean a incorrectly whitelisted message could have enough spam content
to push it to a positive score, right?

No. If you score, say, -100 from whitelisting, you would continue
running spam tests until you ran out of tests, not stop because you went
some threshold below 0.

And if you're system is that bogged down that you REALLY need to stop
scoring at say 6.0 - it's time to upgrade systems. :)

It probably depends on how much mail you process.

Chris L. Franklin wrote:
Back in 2.64 There was the option to have SA stop going though rules
then it hit a max score.
Is there any option for this in 3.X.X ?

No there wasn't such an option in 2.64, that option existed back in SA 2.31, and
was removed from SA 2.40 and higer because it caused problems.

See bugzilla:

There's work to possibly add some short-circuting to sa 3.2.0 or higher. But
even that might not end up as score based. (ie: it might do a short-circuit for
rules like whitelist_from_rcvd)

See bugzilla:

Ps, The system only would need to "process all the rules regardless"
during the loading of the child.

Well, yes and no. This subject comes up a lot. For the record, I favor an
early exit, as you do. But also for the record, it really is more complex
than you make it out to be, and there are several gotchas involved in the
process.

The most interesting points are: exactly where do you process Bayes rules?,
and: this breaks AWL completely, since it is a score averager, and you are
suddenly working with partial scores that have no reliabilty whatever from
an awl point of view. The end result id that if you enable shortcutting you
ALS have to disable awl, or it will probably start doing Really Bad
Things(tm).

Aside from those two major points, there are a number of minor flow
sequencing problems that have to be solved, or at least definitely answered.

It is obvious that you have to process all negative-scoring rules first,
since that is the only way that you can be sure that you have taken
whitelists and the like into account, and your high positive score really IS
high enough to mark this as spam.

However, what if some of those negative-scoring rules are metas? Now you
have to process the meta dependencies before the negative-scoring meta, even
if the dependencies have a positive score. And what if one of the
dependencies is the AWL score? AWL is supposed to run last to give accurate
results, but now it has to run before some arbitrary number of other rules.
But you shouldn't run AWL at all with shortcutting, and here you not only
have to run it, you have to run it early. Do you drop that meta rule, even
though it can contribute a negative score?

Also, net rules are fired off first before any other rules are processed,
and then harvested after most all the rules have been processed. Do you
still want to do that? do you want to hold firing the net rules until
you see if it is going to be tagged as spam by other rules? (Assuming you
don't have a negative meta that is dependent on a net rule!) But if you do
this, and you assume the mail is NT spam, it is going to process ALL
positifve scoring rules (which is essentially all rules) before it starts
the net tests. Now you have completely lost net overlap, and will end up
sitting on your thumb longer before you can dispose of this message.

And what about user rules, or even user changes to rule scores? This can
change the evaluation order from what you would normally do.

There is also a 'priority' field on the rules that determines the order to
run the tests. This undemines the required order to be able to safely bail
early on short circuiting.

Unfortunately, it ain't trivial to get all, or even most all, of this
working in a way that the mathematically inclined would be willing to
consider the results provably correct. And if the results aren't correct,
why bother making them in the first place?

Loren

Loren Wilton wrote:

>>Ps, The system only would need to "process all the rules regardless"
>>during the loading of the child.
>
>>
>
>Well, yes and no. This subject comes up a lot. For the record, I favor an
>early exit, as you do. But also for the record, it really is more complex
>than you make it out to be, and there are several gotchas involved in the
>process.
>
>The most interesting points are: exactly where do you process Bayes rules?,
>and: this breaks AWL completely, since it is a score averager, and you are
>suddenly working with partial scores that have no reliabilty whatever from
>an awl point of view. The end result id that if you enable shortcutting you
>ALS have to disable awl, or it will probably start doing Really Bad
>Things(tm).
>
>Aside from those two major points, there are a number of minor flow
>sequencing problems that have to be solved, or at least definitely answered.
>
>It is obvious that you have to process all negative-scoring rules first,
>since that is the only way that you can be sure that you have taken
>whitelists and the like into account, and your high positive score really IS
>high enough to mark this as spam.
>
>However, what if some of those negative-scoring rules are metas? Now you
>have to process the meta dependencies before the negative-scoring meta, even
>if the dependencies have a positive score. And what if one of the
>dependencies is the AWL score? AWL is supposed to run last to give accurate
>results, but now it has to run before some arbitrary number of other rules.
>But you shouldn't run AWL at all with shortcutting, and here you not only
>have to run it, you have to run it early. Do you drop that meta rule, even
>though it can contribute a negative score?
>
>Also, net rules are fired off first before any other rules are processed,
>and then harvested after most all the rules have been processed. Do you
>still want to do that? do you want to hold firing the net rules until
>you see if it is going to be tagged as spam by other rules? (Assuming you
>don't have a negative meta that is dependent on a net rule!) But if you do
>this, and you assume the mail is NT spam, it is going to process ALL
>positifve scoring rules (which is essentially all rules) before it starts
>the net tests. Now you have completely lost net overlap, and will end up
>sitting on your thumb longer before you can dispose of this message.
>
>And what about user rules, or even user changes to rule scores? This can
>change the evaluation order from what you would normally do.
>
>There is also a 'priority' field on the rules that determines the order to
>run the tests. This undemines the required order to be able to safely bail
>early on short circuiting.
>
>Unfortunately, it ain't trivial to get all, or even most all, of this
>working in a way that the mathematically inclined would be willing to
>consider the results provably correct. And if the results aren't correct,
>why bother making them in the first place?
>
Loren

For starters AWL, white lists and black lists in my option ar ethe worst
things ever. I disable them from the start. If your going to whitelist
some one, why would you want them to even go though SA. (I don't) and if
there blaklisted I don't want them even want the server accepting a
email for me / the user if they are black listed.

And again negative-scoring is useless if u need to write a negative
score you problitly should rethink your positive scoring rules.

All this taking into a account Removing AWL, and negative-scoring. There
are no real problems.

And as a side note about net rules, if your really into using these then
you'll probabliy just want to tune the server not to accept email from
non-RDNS or invaild dns lookups.
-- Chris L. Franklin --

Sat, 22, 2005 at 11:05:07AM -0400, Chris L. Franklin wrote:
For starters AWL, white lists and black lists in my option ar ethe worst
things ever. I disable them from the start. If your going to whitelist
some one, why would you want them to even go though SA. (I don't)

Because a source that regularly sends you legit email, e.g. a
mailing list, might send email that is borderline spammy and the
only thing that tips it back into legitimate territory is the
autowhitelist and bayes based on what YUR users consider ham.

if there blaklisted I don't want them even want the server
accepting a email for me / the user if they are black listed.

There are lots of blacklists and DNSBLs that work best as
contributors, not as absolute yes/no arbiters of what should be
accepted.

And again negative-scoring is useless if u need to write a negative
score you problitly should rethink your positive scoring rules.

I don't understand why you are using SpamAssassin if you really
believe the above.

All this taking into a account Removing AWL, and negative-scoring. There
are no real problems.

And as a side note about net rules, if your really into using these then
you'll probabliy just want to tune the server not to accept email from
non-RDNS or invaild dns lookups.

Masses of legitimate email comes from hosts with no reverse DNS,
incorrect HEL and other borderline or actual RFC violations.

I don't think you have thought this through and I believe that you
would do well to accept some of the wisdom of those that have. If
not, well, try it, and report back as to how well that works out for
you, so that everyone else can see how wrong they are.

PGP SIGNATURE
Version: GnuPG v1.4.1 (GNU/Linux)

7ZxxA3hcXAY1tydPq4FHs2A=
=5X6f
PGP SIGNATURE