Quota
Introduction
IMAP can be used to archive all your messages. This may include 15
apache.org high-traffic mailing lists and a complete collection of all
fun-ppt/fun-pics/fun-movies mails + attachments, your funny colleagues
have sent to you in the last 5 years. ;-)
Reaching a few GiB is possible. That is why setting up a quota might be
necessary. (If you are the sysadmin, you know that sending a daily
message to your users "please clean up your mailbox" is really futile.)
Sooner or later your boss will notice that you are deleting old messages
underhand
Interfaces
see Quota and ImapMailboxRepository
General
Quota objects are responsible for a list of mailboxes. They are not
treated hierarchically but have to be assigned to each mailbox
individually. When a mailbox is created or renamed, it has to inherit
the quota of its parent.
Implementation of quota management could of course be done in a
hierarchy way.
I came to the conclusion that it makes no sense to have user specific
quotas. E.g. user1 has a 100 MB quota for the whole namespace. He is
employee in the buying department. Why shouldn't he store an important
message into #shared.buying, just because he has exceeded his personal
quota? Forcing the buying department to clean up their shared mailbox
not to reach a limit of 1 GiB makes sense.
Root-Mailbox
Quotas are not hierarchy agnostic. quota-object could be responsible
for a selection of mailboxes even across namespaces.
But this makes management of quotas very complex. It may even be
desirable to allow users to setup their own quotas e.g. for their trash
folder. This would also require checking credentials.
The approach is to allow setting a root mailbox for a quota. this
mailbox itself and children are allowed to be added. Users could be
restricted in only being allowed to setup quotas with a root mailbox
they have the "quota management" right for. quotas bound to a root
mailbox are more easy to browse in a management interface.
Many-To-Many
Because quotas are not hierarchy agnostic, they are not inherited
automatically. There are situations where a mailbox may be bound to
several quota object. E.g. #mail.user1.Trash has a quota of 10 MB setup
by user1. #mail.user1 has a quota of 100 MB setup by his local sysadmin.
"#mail" has a quota of 5 GiB and there is a system wide quota of 10 GiB
bound to all mailboxes.
This means "#mail.user1.Trash" is bound to 4 quota objects. If any of
the 4 quotas is exceeded he cannot store a message.
Quotas In JDBC
In a RDBMS each quota object should hold the current usage. It will need
too much time to compute it always on the fly. This means when a message
is appended to a mailbox every quota object bound to this mailbox has to
be updated. A system-wide quota object has to be update every time a
message is appended somewhere
Incrementing an integer value should not be so time intensive.
Transactions will guarantee a consistent state.
This is how I think such a transaction could work:
1. Fetch all quota objects bound to "#mail.user1.Trash" and check
if there is enough space to try the transaction
2. Start transaction
3. Store the header
4. Store the message (this may take long)
5. update the quota objects ( No one else is able to update these
quota objects anymore)
6. Check if we are in the limit (doing a select now should give us
the updated values)
7. Commit if we are in, rollback if we are out
The weak point is that if there is a system-wide quota noone else could
append a message anywhere between 5 and 7. Do we have to worry about
that? Normally this could be done in a few milliseconds. The expensive
parts (storing content) are non-blocking.
To unsubscribe, e-mail: server-dev-unsubscribe (AT) james (DOT) apache.org
For additional commands, e-mail: server-dev-help (AT) james (DOT) apache.org

Joachim,
are you really making that good progress you are already discussing
advanced features, or are quotas required by IMAP?

Quotas should be a generic James concept, not something restricted to
IMAP. As the whole Repository/Mailbox architecture should continue to
not be protocol agnostic.

more inline

Joachim Draeger wrote:

Quota

Introduction

IMAP can be used to archive all your messages. This may include 15
apache.org high-traffic mailing lists and a complete collection of all
fun-ppt/fun-pics/fun-movies mails + attachments, your funny colleagues
have sent to you in the last 5 years. ;-)

:-) ok, quota is a good thing. I'm convinced.

Interfaces

see Quota and ImapMailboxRepository

General

Quota objects are responsible for a list of mailboxes. They are not
treated hierarchically but have to be assigned to each mailbox
individually. When a mailbox is created or renamed, it has to inherit
the quota of its parent.

Implementation of quota management could of course be done in a
hierarchy way.

I came to the conclusion that it makes no sense to have user specific
quotas. E.g. user1 has a 100 MB quota for the whole namespace. He is
employee in the buying department. Why shouldn't he store an important
message into #shared.buying, just because he has exceeded his personal
quota? Forcing the buying department to clean up their shared mailbox
not to reach a limit of 1 GiB makes sense.

Root-Mailbox
Quotas are not hierarchy agnostic. quota-object could be responsible
for a selection of mailboxes even across namespaces.

But this makes management of quotas very complex. It may even be
desirable to allow users to setup their own quotas e.g. for their trash
folder. This would also require checking credentials.

The approach is to allow setting a root mailbox for a quota. this
mailbox itself and children are allowed to be added. Users could be
restricted in only being allowed to setup quotas with a root mailbox
they have the "quota management" right for. quotas bound to a root
mailbox are more easy to browse in a management interface.

+1 for "Quotas are working per Mailbox/UserRepository."

Many-To-Many

Because quotas are not hierarchy agnostic, they are not inherited
automatically. There are situations where a mailbox may be bound to
several quota object. E.g. #mail.user1.Trash has a quota of 10 MB setup
by user1. #mail.user1 has a quota of 100 MB setup by his local sysadmin.
"#mail" has a quota of 5 GiB and there is a system wide quota of 10 GiB
bound to all mailboxes.

This means "#mail.user1.Trash" is bound to 4 quota objects. If any of
the 4 quotas is exceeded he cannot store a message.

Quotas In JDBC

In a RDBMS each quota object should hold the current usage. It will need
too much time to compute it always on the fly. This means when a message
is appended to a mailbox every quota object bound to this mailbox has to
be updated. A system-wide quota object has to be update every time a
message is appended somewhere

If you'd separately store the size of every mailbox folder (and not
recalculate from contained messages), it would not be too expensive I
think to calculate totals by traversing the tree.

Incrementing an integer value should not be so time intensive.
Transactions will guarantee a consistent state.

This is how I think such a transaction could work:

1. Fetch all quota objects bound to "#mail.user1.Trash" and check
if there is enough space to try the transaction
2. Start transaction
3. Store the header
4. Store the message (this may take long)
5. update the quota objects ( No one else is able to update these
quota objects anymore)
6. Check if we are in the limit (doing a select now should give us
the updated values)
7. Commit if we are in, rollback if we are out

Why not do the storing anyway, check afterwards if the quota is
exhausted and flag or lock the mailbox until it has been emptied? seems
much more simple approach yet still sufficient. I don't think a 'quota'
means 'hardly enforced physical limit'.

Also, I'd like to see this independent of any database/JDBC
consideration. that'd be too implementation-centric for now.

Bernd

To unsubscribe, e-mail: server-dev-unsubscribe (AT) james (DOT) apache.org
For additional commands, e-mail: server-dev-help (AT) james (DOT) apache.org

Hi Bernd,

Am Mittwoch, den 05.07.2006, 16:23 +0200 schrieb Bernd Fondermann:

are you really making that good progress you are already discussing
advanced features, or are quotas required by IMAP?

Well, the progress is near to alpha for basic commands. What really is
needed now is starting an Imap capable storage back-end.
As I pointed out in the roadmap draft, quota is IM not a feature that
will be implemented in the near future. But sooner or later it will be
essential.
What I don't want to do is just hacking in a JDBC implementation and
throw everything away when the time has come for the next feature.
If we are aware of what we'll need in the future we could now try to
make the right decisions.

If you'd separately store the size of every mailbox folder (and not
recalculate from contained messages), it would not be too expensive I
think to calculate totals by traversing the tree.

I'm not sure. It could mean adding 10000 values on each append/delete. But DBMS are
optimized to do such things very fast.

Incrementing an integer value should not be so time intensive.
Transactions will guarantee a consistent state.

This is how I think such a transaction could work:

1. Fetch all quota objects bound to "#mail.user1.Trash" and check
if there is enough space to try the transaction
2. Start transaction
3. Store the header
4. Store the message (this may take long)
5. update the quota objects ( No one else is able to update these
quota objects anymore)
6. Check if we are in the limit (doing a select now should give us
the updated values)
7. Commit if we are in, rollback if we are out

Why not do the storing anyway, check afterwards if the quota is
exhausted and flag or lock the mailbox until it has been emptied? seems
much more simple approach yet still sufficient.

Yes, I agree that is is sufficient. But I'm not sure if it is much more
simple.

1. check if one of the quota objects is flagged overquoted
2. Start transaction
3. Store the header
4. Store the message
5. update the total mailbox size
6. commit
7. calculate the occupied space for each quota object
9. set the overquoted flag if needed

It is maybe safer because it does not require locking of central
objects. the mailbox is locked (because of updating size) until the
commit.
So far the Quota
() interface only declares the function getStorageUsage(). Both ways are possible.

I don't think a 'quota'
means 'hardly enforced physical limit'.

I agree. JDBC/RDBMS it is relatively easy, transactions should be
used anyway. In other implementations it might be more difficult, for
example when there are concurrent deliveries.

Also, I'd like to see this independent of any database/JDBC
consideration. that'd be too implementation-centric for now.

It was my intent to show how it could be done with JDBC. How to measure
getStorageUsage() and when to decide that it is overquoted and e.g.
throw an Exception is completely implementation specific.
The approaches for JDBC and e.g. file-based will probably quite
different. There might even be limitations like only one quota per
mailbox or quotas have to be hierarchically.

Joachim

To unsubscribe, e-mail: server-dev-unsubscribe (AT) james (DOT) apache.org
For additional commands, e-mail: server-dev-help (AT) james (DOT) apache.org

Joachim Draeger wrote:
Hi Bernd,

Am Mittwoch, den 05.07.2006, 16:23 +0200 schrieb Bernd Fondermann:


>>are you really making that good progress you are already discussing
>>advanced features, or are quotas required by IMAP?

Well, the progress is near to alpha for basic commands. What really is
needed now is starting an Imap capable storage back-end.

It is great how hard you are working on the IMAP topic. I hope to get a
chance to review it soon!

Also, we have to keep in mind how to integrate your code with the James
codebase. But that's for another thread

As I pointed out in the roadmap draft, quota is IM not a feature that
will be implemented in the near future. But sooner or later it will be
essential.

I consider Quotas quite a big feature for James as a whole, but
"essential"? I don't know probably depends on the user requirements.

BTW, are your propositions based on RFC 2087 or is this another beast?

What I don't want to do is just hacking in a JDBC implementation and
throw everything away when the time has come for the next feature.
If we are aware of what we'll need in the future we could now try to
make the right decisions.

well, this sounds like the waterfall model to me. let's make decisions
when decisions are due, it's impossible to take everything into account
beforehand. And I don't think you'd have to "throw everything away", if
you'd skip thinking about quota now. instead I think one can yield much
better results by concentrating on current tasks.

>>If you'd separately store the size of every mailbox folder (and not
>>recalculate from contained messages), it would not be too expensive I
>>think to calculate totals by traversing the tree.

I'm not sure. It could mean adding 10000 values on each append/delete. But DBMS are
optimized to do such things very fast.

To make this more clear, my proposition was to store only folder sizes
somewhere, never computed sizes. Computed totals could be done using
SELECT (in the DBMS case). Each add/delete would result in changing the
size for exactly one folder.

But it's way to early to optimize unwritten code ;-)

Bernd

To unsubscribe, e-mail: server-dev-unsubscribe (AT) james (DOT) apache.org
For additional commands, e-mail: server-dev-help (AT) james (DOT) apache.org

Am Samstag, den 08.07.2006, 13:41 +0200 schrieb Bernd Fondermann:

>>are you really making that good progress you are already discussing
>>advanced features, or are quotas required by IMAP?

Well, the progress is near to alpha for basic commands. What really is
needed now is starting an Imap capable storage back-end.

It is great how hard you are working on the IMAP topic. I hope to get a
chance to review it soon!

That would be great! At the moment I make only sporadic changes to the
draft API interfaces on SVN.
Things would get more dynamic with external input. :-) It's really
difficult to fit every need and to keep things simple.

Also, we have to keep in mind how to integrate your code with the James
codebase. But that's for another thread

I think a lot about that. I also have some ideas. question is for
example how could James benefit from a logical namespace for message
repositories / mailboxes?
But IM the first solution will be to allow optionally plugging in the
namespace/hierarchy aware repository and using wrappers for legacy code.
(a NamespaceMailRepository implementation).
So the codebase keeps stable.

BTW, are your propositions based on RFC 2087 or is this another beast?

I did it according to RFC 2087 and JavaMail. The procedure I described
as a proposal for JDBC is just a possibility not a requirement.

What I don't want to do is just hacking in a JDBC implementation and
throw everything away when the time has come for the next feature.
If we are aware of what we'll need in the future we could now try to
make the right decisions.

well, this sounds like the waterfall model to me. let's make decisions
when decisions are due, it's impossible to take everything into account
beforehand. And I don't think you'd have to "throw everything away", if
you'd skip thinking about quota now. instead I think one can yield much
better results by concentrating on current tasks.

"Waterfall model" is really getting a swear-word in today's agile
development world, isn't it? ;-)
No waterfall model, just an overview. No complete elaborated plan, just
a few thoughts and drafts. And I promise just to skip thinking about
quota right now, because it should be enough as an overview. :-)

Joachim

To unsubscribe, e-mail: server-dev-unsubscribe (AT) james (DOT) apache.org
For additional commands, e-mail: server-dev-help (AT) james (DOT) apache.org

Joachim Draeger wrote:
Am Samstag, den 08.07.2006, 13:41 +0200 schrieb Bernd Fondermann:

are you really making that good progress you are already discussing
advanced features, or are quotas required by IMAP?

Well, the progress is near to alpha for basic commands. What really is
needed now is starting an Imap capable storage back-end.
>>
>>It is great how hard you are working on the IMAP topic. I hope to get a
>>chance to review it soon!

That would be great! At the moment I make only sporadic changes to the
draft API interfaces on SVN.

I'd more or less stick to your JIRA postings/attachements (as JIRA is
one of our 'official' project resources).

Things would get more dynamic with external input. :-) It's really
difficult to fit every need and to keep things simple.

can't promise anything, but at least it's on my agenda.

>>Also, we have to keep in mind how to integrate your code with the James
>>codebase. But that's for another thread

I think a lot about that. I also have some ideas. question is for
example how could James benefit from a logical namespace for message
repositories / mailboxes?
But IM the first solution will be to allow optionally plugging in the
namespace/hierarchy aware repository and using wrappers for legacy code.
(a NamespaceMailRepository implementation).
So the codebase keeps stable.

namespaces for repos is something which is also going around in my mind
for some time. maybe we should have a separate discussion about it in
near future!


>>BTW, are your propositions based on RFC 2087 or is this another beast?

I did it according to RFC 2087 and JavaMail. The procedure I described
as a proposal for JDBC is just a possibility not a requirement.

What I don't want to do is just hacking in a JDBC implementation and
throw everything away when the time has come for the next feature.
If we are aware of what we'll need in the future we could now try to
make the right decisions.
>>
>>well, this sounds like the waterfall model to me. let's make decisions
>>when decisions are due, it's impossible to take everything into account
>>beforehand. And I don't think you'd have to "throw everything away", if
>>you'd skip thinking about quota now. instead I think one can yield much
>>better results by concentrating on current tasks.

"Waterfall model" is really getting a swear-word in today's agile
development world, isn't it? ;-)

;-) it is - I apologize, didn't want to swear at you ;-)

No waterfall model, just an overview. No complete elaborated plan, just
a few thoughts and drafts. And I promise just to skip thinking about
quota right now, because it should be enough as an overview. :-)

we could go on. but we must keep in mind the whole discussion is
repeated in the future when quotas eventually are reconsidered. ;-)

Bernd

To unsubscribe, e-mail: server-dev-unsubscribe (AT) james (DOT) apache.org
For additional commands, e-mail: server-dev-help (AT) james (DOT) apache.org

Am Montag, den 10.07.2006, 16:18 +0200 schrieb Bernd Fondermann:

are you really making that good progress you are already discussing
advanced features, or are quotas required by IMAP?

Well, the progress is near to alpha for basic commands. What really is
needed now is starting an Imap capable storage back-end.
>>
>>It is great how hard you are working on the IMAP topic. I hope to get a
>>chance to review it soon!

That would be great! At the moment I make only sporadic changes to the
draft API interfaces on SVN.

I'd more or less stick to your JIRA postings/attachements (as JIRA is
one of our 'official' project resources).

I understand that. But as I said before it's always a lot of work to
make a release even if it is only a draft.
I made the effort and brought the interfaces to some kind of a
consistent state and attached it to Jira.

>>Also, we have to keep in mind how to integrate your code with the James
>>codebase. But that's for another thread

I think a lot about that. I also have some ideas. question is for
example how could James benefit from a logical namespace for message
repositories / mailboxes?
But IM the first solution will be to allow optionally plugging in the
namespace/hierarchy aware repository and using wrappers for legacy code.
(a NamespaceMailRepository implementation).
So the codebase keeps stable.
namespaces for repos is something which is also going around in my mind
for some time. maybe we should have a separate discussion about it in
near future!

I hope soon, because it's getting a blocker for my work on imap.

No waterfall model, just an overview. No complete elaborated plan, just
a few thoughts and drafts. And I promise just to skip thinking about
quota right now, because it should be enough as an overview. :-)

we could go on. but we must keep in mind the whole discussion is
repeated in the future when quotas eventually are reconsidered. ;-)

The optimist would say ideas will keep growing until then and we'll be
able to discuss on a higher level. :-)
Well, I agree there are more important topics now.

Joachim

To unsubscribe, e-mail: server-dev-unsubscribe (AT) james (DOT) apache.org
For additional commands, e-mail: server-dev-help (AT) james (DOT) apache.org