Message
From: "Dennis Allison" <allison (AT) shasta (DOT) stanford.edu>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: <zope (AT) zope (DOT) org>
Sent: Tuesday, July 04, 2006 1:27 PM
Subject: Re: [Zope] Trying to trap ConflictError

Johnathan,

I have been lurking, following your thread, but not paying full
attention. Excuse me if thes points have been made before.

There are many long threads on Conflict Errors. You should read them.
Note that there have been significant changes in management strategy
beginning with Zope 2.9.X.

A "Conflict Error" is not really an error in the usual sense; it is an
indication of conflicting transactions in the ZDB, that is, transactions
have violated the optimistic locking used by the ZDB. In general,
Conflict Errors are resolved without problem. The resolution usually
involves rolling back the older transaction and committing the newer
transaction. When the actions associated with the rolled back transaction
are not themselves fully transactional (for example, a file write) there
can be unanticipated side effects.

Your conflicts appear to be in Temporary Storage which is used for
Session Variables. The version that ships with Zope is flexible and
complex and, IMH, overkill. Moreover, it has some unexpected properties,
the most significant is that, even though session variables are stored in
RAM, it occasionally uses the ZDB to manage session variable expiration.
You can often decrease the number of session variable induced conflicts by
increasing the lifetime of session variables.

As both you and Andreas have recommended, I have started researching 'ZDB
Conflict Resolution' and if I come up with generic/reusable external method
application-level solution I will post it.

By way of providing more information:

I am not using sessions. I am using TemporaryStorage to hold small,
custom-made, short-lived image objects (the use case requires that each user
views a static image with a custom overlay, and that the user accesses this
customized image via an html img tag). The custom images are viewed only
once, and then can be deleted. Hence the use of TemporaryStorage (to
eliminate many disk writes and ZDB bloating). The image objects stored in
TemporaryStorage are never overwritten, edited etc (which is causing me some
confusion as to why the conflict errors are occurring), they are simply
created, stored in the TemporaryStorage folder (actually a BTreeFolder2
folder contained within the TemporaryStorage folder) and then 2 minutes
later are deleted. I am guessing that the conflicts are occuring as the
TemporaryStorage is trying to delete old images while new image object 'add
requests' are piling up, but I have yet to determine the cause of the
conflict with 100% certainty (conflicts only occur under heavy loading, and
not always in the same place in the load testing).

Thanks again for everyone's help!

Jonathan

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

Message
From: <dieter (AT) handshake (DOT) de>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: "Dennis Allison" <allison (AT) shasta (DOT) stanford.edu>; <zope (AT) zope (DOT) org>
Sent: Wednesday, July 05, 2006 2:28 PM
Subject: Re: [Zope] Trying to trap ConflictError

Jonathan wrote at 2006-7-4 13:46 -0400:
>
>>The image objects stored in
>>TemporaryStorage are never overwritten, edited etc (which is causing me
>>some
>>confusion as to why the conflict errors are occurring)
>
The error message tells your that the conflict is not caused by
the modification of the image but the container holding your image.

"BTreeFolder" already do conflict resolution -- and reduce the
conflict probability by about a factor of 30 to 100 (depending on type).

However, if several threads should try to add elements with the same
id, then the conflict resolution cannot work.
Maybe, that happens in your case?

The id for the image object is generated by the following code:

def MakeId():
time.sleep(.01)
ts = string.replace(str(time.time()),'.', '') # remove floating point
'.' char
return ts[-10:]

So there should not be any collision of ids.

I am still investing, and have found errors like:

2006-07-04T14:48:12 ERRR ZDB.Connection Couldn't load state for 0x1c
Traceback (most recent call last):
File "/", line 732, in
setstate
self._setstate(obj)
File "/", line 768, in
_setstate
p, serial = self._storage.load(obj._p_oid, self._version)
File "/",
line 104, in load
s=self._index[oid]
KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c'

in the event.log. From my googling research, it appears that these type of
errors may be due to insufficient memory (but I am not sure and am still
investigating to see if this is the cause of the conflict errors).

Thanks to all for the suggestions and assistance!

Jonathan

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

Jonathan wrote:

Message From: <dieter (AT) handshake (DOT) de>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: "Dennis Allison" <allison (AT) shasta (DOT) stanford.edu>; <zope (AT) zope (DOT) org>
Sent: Wednesday, July 05, 2006 2:28 PM
Subject: Re: [Zope] Trying to trap ConflictError
>
>
>Jonathan wrote at 2006-7-4 13:46 -0400:
>>

The image objects stored in
TemporaryStorage are never overwritten, edited etc (which is causing
me some
confusion as to why the conflict errors are occurring)
>>
>>
>The error message tells your that the conflict is not caused by
>the modification of the image but the container holding your image.
>>
>"BTreeFolder" already do conflict resolution -- and reduce the
>conflict probability by about a factor of 30 to 100 (depending on type).
>>
>However, if several threads should try to add elements with the same
>id, then the conflict resolution cannot work.
>Maybe, that happens in your case?
>
>
The id for the image object is generated by the following code:

def MakeId():
time.sleep(.01)
ts = string.replace(str(time.time()),'.', '') # remove floating
point '.' char
return ts[-10:]

So there should not be any collision of ids.

I am still investing, and have found errors like:

Jonathan,

I wouldn't feel safe with that. Why not append a few random digits just
to be safe?

David

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

Message
From: "David H" <bluepaul (AT) earthlink (DOT) net>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: <dieter (AT) handshake (DOT) de>; <zope (AT) zope (DOT) org>
Sent: Wednesday, July 05, 2006 3:00 PM
Subject: Re: [Zope] Trying to trap ConflictError

Jonathan wrote:
>
>>
>Message From: <dieter (AT) handshake (DOT) de>
>To: "Jonathan" <dev101 (AT) magma (DOT) ca>
>Cc: "Dennis Allison" <allison (AT) shasta (DOT) stanford.edu>; <zope (AT) zope (DOT) org>
>Sent: Wednesday, July 05, 2006 2:28 PM
>Subject: Re: [Zope] Trying to trap ConflictError
>>
>>
Jonathan wrote at 2006-7-4 13:46 -0400:

The image objects stored in
TemporaryStorage are never overwritten, edited etc (which is causing me
some
confusion as to why the conflict errors are occurring)

The error message tells your that the conflict is not caused by
the modification of the image but the container holding your image.

"BTreeFolder" already do conflict resolution -- and reduce the
conflict probability by about a factor of 30 to 100 (depending on type).

However, if several threads should try to add elements with the same
id, then the conflict resolution cannot work.
Maybe, that happens in your case?
>>
>>
>The id for the image object is generated by the following code:
>>
>def MakeId():
>time.sleep(.01)
>ts = string.replace(str(time.time()),'.', '') # remove floating point
>'.' char
>return ts[-10:]
>>
>So there should not be any collision of ids.
>>
>I am still investing, and have found errors like:
>>
Jonathan,

I wouldn't feel safe with that. Why not append a few random digits just
to be safe?

I am relatively new to python, so if there is something problematic with my
use of the time module to create unique ids, please let me know! (just
adding random digits seems a bit kludgy?!)

Thanks,

Jonathan

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

Jonathan wrote:

Message From: "David H" <bluepaul (AT) earthlink (DOT) net>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: <dieter (AT) handshake (DOT) de>; <zope (AT) zope (DOT) org>
Sent: Wednesday, July 05, 2006 3:00 PM
Subject: Re: [Zope] Trying to trap ConflictError
>
>
>Jonathan wrote:
>>

Message From: <dieter (AT) handshake (DOT) de>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: "Dennis Allison" <allison (AT) shasta (DOT) stanford.edu>; <zope (AT) zope (DOT) org>
Sent: Wednesday, July 05, 2006 2:28 PM
Subject: Re: [Zope] Trying to trap ConflictError

Jonathan wrote at 2006-7-4 13:46 -0400:

The image objects stored in
TemporaryStorage are never overwritten, edited etc (which is
causing me some
confusion as to why the conflict errors are occurring)

The error message tells your that the conflict is not caused by
the modification of the image but the container holding your image.

"BTreeFolder" already do conflict resolution -- and reduce the
conflict probability by about a factor of 30 to 100 (depending on
type).

However, if several threads should try to add elements with the same
id, then the conflict resolution cannot work.
Maybe, that happens in your case?

The id for the image object is generated by the following code:

def MakeId():
time.sleep(.01)
ts = string.replace(str(time.time()),'.', '') # remove floating
point '.' char
return ts[-10:]

So there should not be any collision of ids.

I am still investing, and have found errors like:

>Jonathan,
>>
>I wouldn't feel safe with that. Why not append a few random digits
>just to be safe?
>
>
I am relatively new to python, so if there is something problematic
with my use of the time module to create unique ids, please let me
know! (just adding random digits seems a bit kludgy?!)

Thanks,

Jonathan

Hi Jonathan,

I figured that if conflicts occur because mutliple processes are
requesting some service at the same time +- delta then simply generating
a file name based on the time was suspect.

David

David

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

Message
From: "David H" <bluepaul (AT) earthlink (DOT) net>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: <zope (AT) zope (DOT) org>
Sent: Wednesday, July 05, 2006 3:22 PM
Subject: Re: [Zope] Trying to trap ConflictError

Jonathan wrote:
>
>>
>Message From: "David H" <bluepaul (AT) earthlink (DOT) net>
>To: "Jonathan" <dev101 (AT) magma (DOT) ca>
>Cc: <dieter (AT) handshake (DOT) de>; <zope (AT) zope (DOT) org>
>Sent: Wednesday, July 05, 2006 3:00 PM
>Subject: Re: [Zope] Trying to trap ConflictError
>>
>>
Jonathan wrote:

Message From: <dieter (AT) handshake (DOT) de>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: "Dennis Allison" <allison (AT) shasta (DOT) stanford.edu>; <zope (AT) zope (DOT) org>
Sent: Wednesday, July 05, 2006 2:28 PM
Subject: Re: [Zope] Trying to trap ConflictError

Jonathan wrote at 2006-7-4 13:46 -0400:

The image objects stored in
TemporaryStorage are never overwritten, edited etc (which is causing
me some
confusion as to why the conflict errors are occurring)

The error message tells your that the conflict is not caused by
the modification of the image but the container holding your image.

"BTreeFolder" already do conflict resolution -- and reduce the
conflict probability by about a factor of 30 to 100 (depending on
type).

However, if several threads should try to add elements with the same
id, then the conflict resolution cannot work.
Maybe, that happens in your case?

The id for the image object is generated by the following code:

def MakeId():
time.sleep(.01)
ts = string.replace(str(time.time()),'.', '') # remove floating
point '.' char
return ts[-10:]

So there should not be any collision of ids.

I am still investing, and have found errors like:

Jonathan,

I wouldn't feel safe with that. Why not append a few random digits just
to be safe?
>>
>>
>I am relatively new to python, so if there is something problematic with
>my use of the time module to create unique ids, please let me know! (just
>adding random digits seems a bit kludgy?!)
>>
>Thanks,
>>
>Jonathan
>>
Hi Jonathan,

I figured that if conflicts occur because mutliple processes are
requesting some service at the same time +- delta then simply generating a
file name based on the time was suspect.

Good thought! I had not considered problems arising from multiple
processes. I have now included a random 3 digit suffix as part of the id
generation routine. the downside, I am still getting conflict errors
(back to the investigation sigh).

Thanks David,

Jonathan

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

7/6/06, Jonathan <dev101 (AT) magma (DOT) cawrote:

I am still investing, and have found errors like:

2006-07-04T14:48:12 ERRR ZDB.Connection Couldn't load state for 0x1c
Traceback (most recent call last):
File "/", line 732, in
setstate
self._setstate(obj)
File "/", line 768, in
_setstate
p, serial = self._storage.load(obj._p_oid, self._version)
File "/",
line 104, in load
s=self._index[oid]
KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c'

in the event.log. From my googling research, it appears that these type of
errors may be due to insufficient memory (but I am not sure and am still
investigating to see if this is the cause of the conflict errors).

TempoaryStorage.py has a few tuning constants. The above error reads
as though RECENTLY_GCIDS_LEN is too small for your application. It
may well be that the default tuning of TempoaryStorage does not match
your application very well.

It would be useful to try FileStorage to help decide on where to focus
attention. If FileStorage behaves a lot better then that also suggests
that TempoaryStorage is miss-tuned for your application.

Cheers
Michael

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

Message
From: <dieter (AT) handshake (DOT) de>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: "Dennis Allison" <allison (AT) shasta (DOT) stanford.edu>; <zope (AT) zope (DOT) org>
Sent: Wednesday, July 05, 2006 3:25 PM
Subject: Re: [Zope] Trying to trap ConflictError

Jonathan wrote at 2006-7-5 14:40 -0400:
>
>>The id for the image object is generated by the following code:
>>
>>def MakeId():
>time.sleep(.01)
>ts = string.replace(str(time.time()),'.', '') # remove floating point
>>'.' char
>return ts[-10:]
>>
>>So there should not be any collision of ids.
>
You should verify this. I doubt that you are right, as both the
scheduling as well as the clock granularity are discrete (and rather
coarse).

I suggest you use either the mechanism of

* the "BrowserIdManager"::it combines a time with a random number

* the "ZCatalog": it chooses per thread a random number and
then assignes sequentially, until a conflict happens.
Then it restarts (choosing a random number)
>
>>I am still investing, and have found errors like:
>>
>>2006-07-04T14:48:12 ERRR ZDB.Connection Couldn't load state for 0x1c
>>Traceback (most recent call last):
>File "/", line 732, in
>>setstate
>self._setstate(obj)
>File "/", line 768, in
setstate
>p, serial = self._storage.load(obj._p_oid, self._version)
>File "/",
>>line 104, in load
>s=self._index[oid]
>>KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c'
>>
>>in the event.log.
>
Apparently, you have more than a single type of error
--
Are you the person that used "try: except: "?

The non-deterministic inconsistencies caused by such constructs
can look like what you decribe above

I was originally using the try/except to trap the conflict error, but as a
result of being educated as to the futility of that approach I have since
removed the try/except.

I have also included a random 3 digit suffix in the id generation routine to
address the issue of muliple processes potentially generating the same id.
However, the conflict errors remain, so I am back to investigating

Thanks for the help,

Jonathan

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

Message
From: "Michael Dunstan" <michael.dunstan (AT) gmail (DOT) com>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: <zope (AT) zope (DOT) org>
Sent: Wednesday, July 05, 2006 5:15 PM
Subject: Re: [Zope] Trying to trap ConflictError

TempoaryStorage.py has a few tuning constants. The above error reads
as though RECENTLY_GCIDS_LEN is too small for your application. It
may well be that the default tuning of TempoaryStorage does not match
your application very well.

It would be useful to try FileStorage to help decide on where to focus
attention. If FileStorage behaves a lot better then that also suggests
that TempoaryStorage is miss-tuned for your application.

Thanks for the idea Micheal as a test I changed from TemporaryStorage to
FileStorage to see if the ConflictErrors would 'go away'. However, the
ConflictErrors still occur! This leads me to believe that the problem is
not related to TemporaryStorage.

Further investigation led to the discovery that ConflictErrors are being
raised by TemporaryStorage and FileStorage in the same 'store' routines.
Here is the relevant code from TemporaryStorage:

def store(self, oid, serial, data, version, transaction):
if transaction is not self._transaction:
raise (self, transaction)
self._lock_acquire()
try:
if self._index.has_key(oid):
oserial=self._index[oid]
if serial != oserial:
newdata = self.tryToResolveConflict(
oid, oserial, serial, data)
if not newdata:
raise PSException.ConflictError(
oid=oid,
serials=(oserial, serial),
data=data) # Conflict Error raised
here
else:
data = newdata
else:
oserial = serial
newserial=self._tid
self._tmp.append((oid, data))
return serial == oserial and newserial or ResolvedSerial
finally:
self._lock_release()


>From what I understand from the above code, the error is being raised
because the oid already exists and the object pointed to by the old oid is
different from the object pointed to by the new oid, hence the conflict
error.

I haven't been able to determine how the oid's are generated, other than
"little-endian 64-bit unsigned integers that will be assigned more or less
sequentially", but I can't figure out how the same oid is being used twice
(on the other hand I could be totally off-base as I am really way over my
head here!).

again, any and all ideas, comments, suggestions are greatly
appreciated!

Jonathan

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

7/7/06, Jonathan <dev101 (AT) magma (DOT) cawrote:
I haven't been able to determine how the oid's are generated, other than
"little-endian 64-bit unsigned integers that will be assigned more or less
sequentially", but I can't figure out how the same oid is being used twice
(on the other hand I could be totally off-base as I am really way over my
head here!).

An oid is an identifier for a persistent state of an object. As the
state of an object evolves from transaction to transaction it gets a
new oid for each state. Two concurrent transactions that involve the
same object will start with the same state, i.e. the same oid, for
that object.

Some detail about how BTrees behave can be found in

Also might be illustrative to run fsdump to see what is happening in
your transactions. See
You'll need
FileStorage for that. And that only shows you details of successful
transactions.

cheers
michael

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

7/7/06, Michael Dunstan <michael.dunstan (AT) gmail (DOT) comwrote:
An oid is an identifier for a persistent state of an object. As the
state of an object evolves from transaction to transaction it gets a
new oid for each state. Two concurrent transactions that involve the
same object will start with the same state, i.e. the same oid, for
that object.

Sorry. Ignore that. :-(

Note to self, don't even read email before morning caffeine intake.

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)

Message
From: <dieter (AT) handshake (DOT) de>
To: "Jonathan" <dev101 (AT) magma (DOT) ca>
Cc: "Michael Dunstan" <michael.dunstan (AT) gmail (DOT) com>; <zope (AT) zope (DOT) org>
Sent: Thursday, July 06, 2006 2:56 PM
Subject: Re: [Zope] Trying to trap ConflictError

Jonathan wrote at 2006-7-6 09:53 -0400:

From what I understand from the above code, the error is being raised
>>because the oid already exists and the object pointed to by the old oid is
>>different from the object pointed to by the new oid, hence the conflict
>>error.
>
You interpretation is slightly wrong. What really differs is not
the object (or "oid") but the timestamp.

Each persistent object has a timestamp "_p_serial" which
identifies the transaction that made the last change to the object.

When an object is loaded from the ZDB, its timestamp is
recorded (in "_p_serial"). When later a change to the object
should be commited, the recorded timestamp is compared to the object's
current timestamp in the ZDB. If the timestamps differ, then
the object was modified by a concurrent transaction and our
modification is likely to be based on stale information.
As a result a "ConflictError" is raised.

Thanks Dieter, this got me pointed in the right direction

The problem was due to multiple processes simultaneously accessing a
'MakeId' routine (as some had suggested earlier in this investigation!).
Even though the routine returned a 10 digit id consisting of a 7 digit
timestamp + 3 random digits, there were still id collisions.

Changing the MakeId routine to return a 10 character string (all 10
characters randomly generated) eliminated the ConflictErrors that were
occurring during heavy load testing.

Thanks to everyone for the great assistance!!!

Jonathan

Zope maillist - Zope (AT) zope (DOT) org

** No cross posts or HTML encoding! **
(Related lists -

)