I have just had a small problem with a user complaining about email
delays and tracked it down to the
parameter, which has a default value of 10.
After 10 messages are injected via SMTP on the same
connection, further messages are placed in the queue
rather than being handled immediately.
Now, I know it's all documented etc, but the idea of default values
is they work in the majority of cases. This value seems very low
to me for a default value. After all, if the client had made
multiple connections all would have been fine, except my server would
have been under more load and it would have been harder to
do rate limiting. I think I want to encourage multiple
messages on the same connection, not discourage them.
Can someone explain the rationale behind such a low value?
It would seem to me that treating messages delivered in the
same connection differently to those arriving
on different connections is not actually that helpful
but maybe I just have not thought this through enough.
Regards,
pdg

Thu, 17 Aug 2006, Peter D. Gray wrote:

After 10 messages are injected via SMTP on the same
connection, further messages are placed in the queue
rather than being handled immediately.

<snip>

Can someone explain the rationale behind such a low value?

The rationale for having a limit is that otherwise Exim starts a
delivery process for each message as soon as it arrives, and if 1000
arrive quickly you would suddenly have 1000 delivery processes all
trying to run at once.

It would seem to me that treating messages delivered in the
same connection differently to those arriving
on different connections is not actually that helpful
but maybe I just have not thought this through enough.

You can limit the number of different connections. If you have set that
to, say, 100, and messages come in very fast over those 100, you might
have (with the default setting) 1000 deliveries going on simultaneously,
but no more. Without the limit on each individual connection the number
could get a lot higher. Agreed, it's unlikely that all 100 connections
will be spewing messages in this fast, but some limit is needed IMH

As to the value of 10, well, yes, it's small. Partly that was me being
conservative, and partly it's because it was invented some time ago
(1998) when machines were less powerful, and partly it's because most
connections only send one message anyway

Thu, Aug 17, 2006 at 10:34:59AM +0100, Philip Hazel wrote:
Thu, 17 Aug 2006, Peter D. Gray wrote:

After 10 messages are injected via SMTP on the same
connection, further messages are placed in the queue
rather than being handled immediately.

<snip>

Can someone explain the rationale behind such a low value?

The rationale for having a limit is that otherwise Exim starts a
delivery process for each message as soon as it arrives, and if 1000
arrive quickly you would suddenly have 1000 delivery processes all
trying to run at once.

Yes, the limit is a good idea.

It would seem to me that treating messages delivered in the
same connection differently to those arriving
on different connections is not actually that helpful
but maybe I just have not thought this through enough.

You can limit the number of different connections. If you have set that
to, say, 100, and messages come in very fast over those 100, you might
have (with the default setting) 1000 deliveries going on simultaneously,
but no more. Without the limit on each individual connection the number
could get a lot higher. Agreed, it's unlikely that all 100 connections
will be spewing messages in this fast, but some limit is needed IMH

As to the value of 10, well, yes, it's small. Partly that was me being
conservative, and partly it's because it was invented some time ago
(1998) when machines were less powerful, and partly it's because most
connections only send one message anyway

I think you should at least consider making the limit 0 by default.

It seems to me that most (in fact nearly all)
of the "rate limiting" settings
in exim are turned off (eg queue_only_load). The assumption
seems to be that if you want/need rate limiting, you turn it
on yourself. The default value for
seems a notable exception and may catch people out. It certainly
caught me. As MTA's get smarter, they tend to batch things
more and more.

Just my 2c worth.

And thank you for the prompt reply.

Regards,
pdg

Fri, 18 Aug 2006, Peter D. Gray wrote:

I think you should at least consider making the limit 0 by default.

I don't like incompatible changes. The limit was introduced for a reason
(somebody got bitten). I would need a lot of convincing by a lot of
people before I'd feel that it could be changed.

It seems to me that most (in fact nearly all) of the "rate limiting"
settings in exim are turned off (eg queue_only_load).

What about these (they are just a few that I found with a quick grep):

|smtp_accept_max|Use: main|Type: integer|Default: 20|
|smtp_connect_backlog|Use: main|Type: integer|Default: 20|
|smtp_max_synprot_errors|Use: main|Type: integer|Default: 3|
|smtp_max_unknown_commands|Use: main|Type: integer|Default: 3|

You are right, though, that there is inconsistency. It's hard to be
consistent. In the case of queue_only_load, I think it might be hard to
pick a number that would be suitable for the huge variety of hardware
configurations that Exim runs on. Maybe. I'm no expert in exactly what
the load value means.

Fri, 18 Aug 2006, Peter D. Gray wrote:

I think you should at least consider making the limit 0 by default.

I don't like incompatible changes. The limit was introduced for a reason
(somebody got bitten). I would need a lot of convincing by a lot of
people before I'd feel that it could be changed.

It seems to me that most (in fact nearly all) of the "rate limiting"
settings in exim are turned off (eg queue_only_load).

What about these (they are just a few that I found with a quick grep):

|smtp_accept_max|Use: main|Type: integer|Default: 20|
|smtp_connect_backlog|Use: main|Type: integer|Default: 20|
|smtp_max_synprot_errors|Use: main|Type: integer|Default: 3|
|smtp_max_unknown_commands|Use: main|Type: integer|Default: 3|

You are right, though, that there is inconsistency. It's hard to be
consistent. In the case of queue_only_load, I think it might be hard to
pick a number that would be suitable for the huge variety of hardware
configurations that Exim runs on. Maybe. I'm no expert in exactly what
the load value means.

Fri, Aug 18, 2006 at 09:26:31AM +0100, Philip Hazel wrote:
Fri, 18 Aug 2006, Peter D. Gray wrote:

I think you should at least consider making the limit 0 by default.

I don't like incompatible changes. The limit was introduced for a reason
(somebody got bitten). I would need a lot of convincing by a lot of
people before I'd feel that it could be changed.

That's ok. As I said, I think you should "consider it". If you have
reasons for leaving it the way it is then that's fine as well.

the issue of compatibility though, you are between a rock
and a hard place. Compatability is good, but things working
right is better (IMH). Consistancy is also good.
You have to juggle these competing demands.

It seems to me that most (in fact nearly all) of the "rate limiting"
settings in exim are turned off (eg queue_only_load).

What about these (they are just a few that I found with a quick grep):

|smtp_accept_max|Use: main|Type: integer|Default: 20|
|smtp_connect_backlog|Use: main|Type: integer|Default: 20|
|smtp_max_synprot_errors|Use: main|Type: integer|Default: 3|
|smtp_max_unknown_commands|Use: main|Type: integer|Default: 3|

You are right, though, that there is inconsistency. It's hard to be
consistent. In the case of queue_only_load, I think it might be hard to
pick a number that would be suitable for the huge variety of hardware
configurations that Exim runs on. Maybe. I'm no expert in exactly what
the load value means.

You are correct. It is becoming increasing hard to work out what the
load average is telling you, even though the definition is very clear.
I have found recently that machines seem to have higher load averages
that I would have predicted from their job mix and they seem to work
well with high loads (solaris/sparc).

Regards,
pdg

Fri, Aug 18, 2006 at 09:26:31AM +0100, Philip Hazel wrote:
Fri, 18 Aug 2006, Peter D. Gray wrote:

I think you should at least consider making the limit 0 by default.

I don't like incompatible changes. The limit was introduced for a reason
(somebody got bitten). I would need a lot of convincing by a lot of
people before I'd feel that it could be changed.

That's ok. As I said, I think you should "consider it". If you have
reasons for leaving it the way it is then that's fine as well.

the issue of compatibility though, you are between a rock
and a hard place. Compatability is good, but things working
right is better (IMH). Consistancy is also good.
You have to juggle these competing demands.

It seems to me that most (in fact nearly all) of the "rate limiting"
settings in exim are turned off (eg queue_only_load).

What about these (they are just a few that I found with a quick grep):

|smtp_accept_max|Use: main|Type: integer|Default: 20|
|smtp_connect_backlog|Use: main|Type: integer|Default: 20|
|smtp_max_synprot_errors|Use: main|Type: integer|Default: 3|
|smtp_max_unknown_commands|Use: main|Type: integer|Default: 3|

You are right, though, that there is inconsistency. It's hard to be
consistent. In the case of queue_only_load, I think it might be hard to
pick a number that would be suitable for the huge variety of hardware
configurations that Exim runs on. Maybe. I'm no expert in exactly what
the load value means.

You are correct. It is becoming increasing hard to work out what the
load average is telling you, even though the definition is very clear.
I have found recently that machines seem to have higher load averages
that I would have predicted from their job mix and they seem to work
well with high loads (solaris/sparc).

Regards,
pdg

Fri, 18 Aug 2006, Philip Hazel wrote:

In the case of queue_only_load, I think it might be hard to pick a
number that would be suitable for the huge variety of hardware
configurations that Exim runs on. Maybe. I'm no expert in exactly what
the load value means.

Traditionally it's the average number of CPU-limited and disk-limited
processes. The right load limit setting depends a lot on what your
workload is. If your main bottleneck is disk, then you don't want your
load to get up near one. (You need some headroom to recover from bursts.)
If your main bottleneck is CPU then loads up to the number of cores are
K.

So in the situation where a lot of messages are being pumped down the same
connection, a lot of delivery processes will be fired off concurrently,
which implies a spike in disk load. Queue runner processes, on the other
hand, run serially, so are smoother on the load. For stuff like
SpamAssassin, you can manage the imposed load by tuning the number of
spamd children to some small multiple of the number of CPUs (allowing for
network test latencies), with a suitable connection backlog limit to allow
for burstiness.

Tony.