Wed, May 17, 2006 19:43:14 PM +0200, Ingo K
(kloecker (AT) kde (DOT) org) wrote:
Am Mittwoch, 17. Mai 2006 01:17 schrieb Gurvan Huiban:
Tuesday 16 May 2006 19:17, Ingo K wrote:
Actually, I was thinking "will it be possible to use procmail to
filter-and-deliver emails in the different folders we use with
kmail", not only in a spool folder. I would personally find it
useful.
I don't think we'll make sure that this will be possible with
Akonadi. IW it will probably not work with procmail out of the
box, but maybe it will be possible to write a small helper
application to make procmail able to put messages into different
folders in Akonadi.
IW, when a current Kmail user will upgrade to any KDE-4 based Linux
distro, not only any procmail recipes or any _other_ email
processing/search utility (Perl & shell scripts, what about Beagle?)
would stop working until/unless such "helpers" are made available by
3rd parties?
TIA,
Marco

There are only two options: Either you want fast access to your
information (email, calendar, addresses, you-name-it) which is only
possible through indexes or you want text files which need to be
re-indexed each time an external application (like procmail) messed
around with them invalidating all your index files. We have chosen fast
access.

Let's be reasonable and restrict ourselves to the maildir format :o)

Each email is stored in a file. Correct me if I am wrong, but detecting a
change (email added/removed) in a maildir folder can be done by comparing the
file list of the folder, no? A Kind of
diff `ls maildir_folder` old_index_of_email
gives a list of added/removed files, and consequently emails. I have just
tried the command, there is a problem with the syntax, but you get the point
I hope :o)

How long does such command take on a large email folder? Would it slow down
kmail so much? Moreover, such process could be done in a threaded
asynchronous way (Each 5 seconds, a thread checks the differences in a
folder). Would that be CPU-consuming?

Another directions could be to use one of the "File monitoring system"
(gamin/fam) to alert Kmail when there is a change in an email folder.

I did not study the problem. I guess there are some difficulties I do not
imagine. However, the implementation of a new API could be an occasion to
solve which is (in my opinion) one of the major-and-older flaws with Kmail.

Thursday 18 May 2006 17:56, Gurvan Huiban wrote:
Each email is stored in a file. Correct me if I am wrong, but detecting a
change (email added/removed) in a maildir folder can be done by comparing
the file list of the folder, no? A Kind of

of course. "can do" and "good idea" are very different, however.

How long does such command take on a large email folder? Would it slow down
kmail so much?

yes. since you then have to not only have a watch on the directory, but then
read the file in -after- the initial I/ causing yet more I/ (which is the
most expensive thing one can do on a computer system these days usually) and
the rebuild the index from there. it's much more efficient to index as data
arrives, particularly when you are dealing with 100s or 1000s of pieces of
data (think of shared calendaring, for instance) on a machine used for other
things as well (e.g. web browsing, IM, media playing, etc. the machine isn't
dedicated to PIM stuff. one the most annoying things about tools like beagle
and kat is that they chew up CPU as if they owned the box)

but the -real- problem comes in synchronizing access. if process A can write
while process B may be reading if process C can be traversing the index
while process D is changing the data the index represents it's a mess.
very complex and nigh impossible to get right in a sufficiently complex
system. it may work for a simple mailbox but breaks down completely in even
moderately complex scenarios.

the experience with kontact + kopete + kalarmd + is what made this lesson
so clear to the KDE PIM team (please correct me if i'm wrong there PIM people
=)

so this is why relational database systems tend to have servers that serve as
a single point of access to the data: you can then guarantee atomicity and
consistency in both read and write over multiple simultaneous processes as
well as ensure performance optimizations.

it's the way to do it.

Moreover, such process could be done in a threaded
asynchronous way (Each 5 seconds, a thread checks the differences in a
folder). Would that be CPU-consuming?

yes. threads don't make CPU or sycronizing issues go away; threads are
actually -more- expensive and -increase- sync issues. they simply move
computation away from other bits of computation, allowing (when used
correctly) the illusion of greater interactivity and possibly the reality of
simpler code paths (again, when used correctly, which is sadly often not the
case in practice)

so how does one solve the procmail problem? simple. provide a command line
tool that allows one to interact with akonadi and have procmail use that app
to submit data into the repository. voila.

Thu, May 18, 2006 22:50:02 PM +0200, Ingo K
(kloecker (AT) kde (DOT) org) wrote:

The whole point of Akonadi is that it's a centralized storage service.

Thanks for your explanation. Looking at it, maybe the central question
for me becomes "K, is this centralized service designed from the
ground up to integrate easily, without duplications and reinventing
the wheel, with other non-{KDE, KDEPIM} software, especially that
which already exists?"

I don't know about Beagle, but the KDE equivalent will of course be
able to search email using Akonadi's API.

This is a perfect example of what is worrying me: Beagle is already
around, even if not stable, and can(will) index everything on the
disk, not just email. Will the Akonadi messages be searchable by
Beagle from the beginning?

it will look like a step backward: today (this year) you can
do one search on all your desktop with Beagle; when Akonadi arrives,
you go back to *two* searches, one inside what you put into Akonadi,
another one on everywhere else with Beagle, because bridges between
the two will arrive only as an afterthought, by some 3rd party.

There are only two options: Either you want fast access or you
want text files which need to be re-indexed each time We have
chosen fast access.

After your explanatin I understand this, agree with the principle and
am grateful for your work. I'm (still) only worried that this may
happen without real integration with the rest of the FSS desktop,
that such a thing will have to be added as a patch in a second moment.

Ciao,
Marco

Friday 19 May 2006 07:58, M. Fioretti wrote:

Thanks for your explanation. Looking at it, maybe the central question
for me becomes "K, is this centralized service designed from the
ground up to integrate easily, without duplications and reinventing
the wheel, with other non-{KDE, KDEPIM} software, especially that
which already exists?"

The answer is: Yes. It's one of our explicit design goals.

This is a perfect example of what is worrying me: Beagle is already
around, even if not stable, and can(will) index everything on the
disk, not just email. Will the Akonadi messages be searchable by
Beagle from the beginning?

Beagle support for KMail mails currently is far from optimal. Akonadi will
provide all what is needed to change that. It even will not be much work. So
if Beagle is able to index Akonadi from the beginning is only a question of
somebody writing the corresponding Beagle plugin, which should be simple.
Would you be interested in doing that? By the way, with this you get Beagle
indexing of KDE addressbooks and calendars for free.

Friday 19 May 2006 12:16, Gurvan Huiban wrote:
Finally (correct me if I am wrong), the softwares that allow the user to
use procmail (mutt, some imap server, and so on) decided to choose the
"other" solution to avoid synchronization problems (that is not to rely on
an index file and reparsing the folder/mailbox each time it is open). Is
that correct?

most of these tools were written without any thought put to interoperability,
really. and most importantly, most of these tools do not have syncronization
issues and/or the requirement to allow fast random access.

It would answer the "why mutt can and Kmail can't"? ;o)

in the case of mutt it's a bit simpler since mutt figures that just mutt is
dealing with the data. in kde we have kmail, korganizer, kalarmd, kopete,
kaddressbook, etc, etc, etc it's a much more complex environment.

Fri, May 19, 2006 18:02:34 PM +0200, Ingo K
(kloecker (AT) kde (DOT) org) wrote:

Am Freitag, 19. Mai 2006 06:28 schrieb Aaron J. Seigo:
so how does one solve the procmail problem? simple. provide a
command line tool that allows one to interact with akonadi and
have procmail use that app to submit data into the
repository. voila.
Which is exactly what I already wrote much earlier in this
discussion, but this answer didn't seem to be satisfying. Sigh.

As far as I am concerned, your earlier answer _had_ already been
clear.

My request of clarification was meant to understand exactly _who_, in
the developers vision, is supposed to "provide a command line tool
that allows one to interact with akonadi":

1) the Akonadi developers, as in "we, who did this new cool and
_different_ thing and know best how it works, also took care of
making it easy to test and integrate gradually with tons of things
already in use"

or

2) everybody else (end users, distro packagers, other developers) but
the Akonadi developers, since they already did Akonadi and it's not
their responsibility to sync/integrate with everything else which
was already in use.

*If* I understand correctly Aaron's answer and yours, it's 2), isn't
it? If so, K, I can hack a script or two if I really want to try
akonadi myself some day. I'm not asking or even suggesting that you
do one or the other, I just wanted to know how things stand, really.

Ciao,
Marco

Saturday 20 May 2006 10:29, M. Fioretti wrote:
My request of clarification was meant to understand exactly _who_, in
the developers vision, is supposed to "provide a command line tool
that allows one to interact with akonadi":

*If* I understand correctly Aaron's answer and yours, it's 2), isn't
it? If so, K, I can hack a script or two if I really want to try
akonadi myself some day. I'm not asking or even suggesting that you
do one or the other, I just wanted to know how things stand, really.

i think it's just too early in akonadi development to be able to predict this
sort of thing ("who will write tool X"). what is probably safe to say,
however, is that if there is a need / want for it it will happen. by whose
hands? probably whoever gets to it first depending on the need and their
relation to that need. it's how open source tends to work =)