Mon, Jan 02, 2006 at 11:36:11AM +0200, Tsirkin Evgeny wrote:
Thanks for answer.However i have already searched for a way to make count
faster and didn't find anything.
Any pointers will be appreciated.

What you want to do in the trigger is insert a new row into a table that
contains the change in count, instead of trying to update a single row
for each value/ident (btw, you'll probably get better performance if you
make ident an int instead of a numeric). So now you'll have a list of
changes to the value, which you will periodically want to roll up into a
table that just stores the count.

Wed, 28 Dec 2005, Bruno Wolff III wrote:

Tue, Dec 27, 2005 at 11:48:55 +0200,
Tsirkin Evgeny <tsurkin (AT) mail (DOT) jct.ac.ilwrote:

Hi list!
My issue is as follows :
I have to do somecalculations based on *exact* number of rows in 2
tables (with a filter) meaning:
SELECT count(*) FRM a WHERE a.row1 in (1,2,3,4);
SELECT count(*) FRM b WHERE b.row1 in (1,2,3,4);
However i couldn't use the count(*) since it is too slow beacause of the
table size.

You should look through the past archives on this subject. There is a way to
do this that uses MVCC for incremental changes. You do want to make a sweep
through the delta table periodically which will need stronger locking, but
this won't block reads on a and b.

So,i created a trigger that on insert increments and on delete
decriments special "counter" table
that contains
\d counter
Column | Type | Modifiers
++
counter_type | character varying(30) | the table name
ident | numeric(10,0) |
count | integer
--
The problem of course is thelocking issues while changing a and b
tables.What i am doing now is to
select 1 from counterwhere counter_type='a' and ident in (1,2,3,4)
for update;
select 1 from counterwhere counter_type='b' and ident in (5,6,7) for
update;
Befor changing anything in tables "a" and "b"in transaction .I am also
doing for update select on the
"a" and "b" tables itself ,that is:
select b from awhere pkey in (5,6,7) for update;

My problems:
[1] Is the for update lock anouthhere?
[2] The "for update" queriesHAVE to be done in the same order in all
application which is pretty error prone -
it is very easy to forget in one place and get a deadlock.
[3] Can i make a trigger that automatically locks the counter_type='b'
if a for update select was done on table b?
something like (pseudo):
trigger on select for update table b{
select for update where ident = LD.pkey ;
}
[4] Can i combine queries for multiple tables to make locking atomic:
select 1 from b,counter where b.pkey in (1,2,3,4) and counter.ident in
(1,2,3,4);
Hope for help and sorry for long message.
evgeny
>
>
>
(end of broadcast)
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

(end of broadcast)
TIP 5: don't forget to increase your free space map settings

(end of broadcast)
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly

Evgeny.

Tue, 3 Jan 2006, Jim C. Nasby wrote:

Mon, Jan 02, 2006 at 11:36:11AM +0200, Tsirkin Evgeny wrote:
Thanks for answer.However i have already searched for a way to make count
faster and didn't find anything.
Any pointers will be appreciated.

What you want to do in the trigger isinsert a new row into a table that
contains the change in count, instead of trying to update a single row
for each value/ident (btw, you'll probably get better performance if you
make ident an int instead of a numeric).

Why?

So now you'll have a list of
changes to the value, which you will periodically want to roll up into a
table that just stores the count.
Interesting idea.Thanks.However it pretty complicates things ,maybe there
will be simpler solution.Something i did not thought about at all - i think that
counting is something that everybody does.
[1] I have also a hope that i can create a trigger that locks counter
table once a 'select for update' was done on one of the tables i count.
However how can i say if a select that fires a trigger is a 'for update'
one?
[2] Maybe there is a MVCC or something solution like Bruno suggested (that
i did not realy understood thought).
Evgeny.

Wed, 28 Dec 2005, Bruno Wolff III wrote:

Tue, Dec 27, 2005 at 11:48:55 +0200,
Tsirkin Evgeny <tsurkin (AT) mail (DOT) jct.ac.ilwrote:

Hi list!
My issue is as follows :
I have to do somecalculations based on *exact* number of rows in 2
tables (with a filter) meaning:
SELECT count(*) FRM a WHERE a.row1 in (1,2,3,4);
SELECT count(*) FRM b WHERE b.row1 in (1,2,3,4);
However i couldn't use the count(*) since it is too slow beacause of the
table size.

You should look through the past archives on this subject. There is a way to
do this that uses MVCC for incremental changes. You do want to make a sweep
through the delta table periodically which will need stronger locking, but
this won't block reads on a and b.

So,i created a trigger that on insert increments and on delete
decriments special "counter" table
that contains
\d counter
>Column | Type | Modifiers
++
counter_type | character varying(30) | the table name
ident | numeric(10,0) |
count | integer
--
The problem of course is thelocking issues while changing a and b
tables.What i am doing now is to
select 1 from counterwhere counter_type='a' and ident in (1,2,3,4)
for update;
select 1 from counterwhere counter_type='b' and ident in (5,6,7) for
update;
Befor changing anything in tables "a" and "b"in transaction .I am also
doing for update select on the
"a" and "b" tables itself ,that is:
select b from awhere pkey in (5,6,7) for update;

My problems:
[1] Is the for update lock anouthhere?
[2] The "for update" queriesHAVE to be done in the same order in all
application which is pretty error prone -
it is very easy to forget in one place and get a deadlock.
[3] Can i make a trigger that automatically locks the counter_type='b'
if a for update select was done on table b?
something like (pseudo):
trigger on select for update table b{
>select for update where ident = LD.pkey ;
}
[4] Can i combine queries for multiple tables to make locking atomic:
select 1 from b,counter where b.pkey in (1,2,3,4) and counter.ident in
(1,2,3,4);
Hope for help and sorry for long message.
evgeny
>
>
>
(end of broadcast)
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

(end of broadcast)
TIP 5: don't forget to increase your free space map settings
>
>
>
(end of broadcast)
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly
--

I'm not really sure What Bruno answered that you didn't ask, but I'll
mention that anything you try here other than doing inserts and deletes
from your trigger is flat out going to perform poorly due to locking.
Plus, as you discovered, deadlocking will be a major issue, unless you
establish a table-level lock on your count table at the begining of the
transaction.

Personally, I'd just put the effort into setting up the insert/delete
stuff and a means to periodically roll that info up into a summary
table. I know that this has come up in the past, so you can probably
find someone else who's already done this and might be able to provide
you with code. Even if you have to write it from scratch, I doubt it's
more work than what you've already put into trying to get the other
scheme to work.

BTW, there is a desire to be able to store some kind of visibility info
in a manner which could be used by indexes; that might allow your select
count(*)'s to perform adequately without maintaining a seperate summary
table.

Thu, Jan 05, 2006 at 10:53:40AM +0200, Tsirkin Evgeny wrote:

Bruno,thanks for the answer but this was not at ALL that i asked.
Neither mind .
Note however, how polite i am .
Thanks anyway.
Evgeny

Bruno Wolff III wrote:

Thu, Jan 05, 2006 at 10:34:31 +0200,
Tsirkin Evgeny <tsurkin (AT) mail (DOT) jct.ac.ilwrote:

>
>>I understand THAT ,I meant why int will give me more performance.
>
>>
>
>Because if you have a bunch of processes sitting around waiting for table
>locks, the average time to process queries will be higher. If you are
>already limited by I throughput or CPU usage this may not be a big deal,
>but in many cases it will be.
>

>
>>Well i D have a lot of concurrent queries,that is the whole point.
>
>>
>
>Then you probably want to do what people have been recommending you do.
>

>
>>That does not help me .What i want is :
>>select 1 from table a where whatever for update;
>>issuing this should fire a trigger that does:
>
>>
>
>You can't trigger on select statements. So this approach won't workin any
>case.
>

>
>>select 1 from table counter where a.pkey=counter.ident or something
>>however this should only be done for select FR UPDATE!
>>And i can't know that inside the trigger!
>
>>
>
>(end of broadcast)
>TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly

(end of broadcast)
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Sun, Jan 08, 2006 at 10:57:21AM +0200, Tsirkin Evgeny wrote:

Jim C. Nasby wrote:

>Personally, I'd just put the effort into setting up the insert/delete
>stuff and a means to periodically roll that info up into a summary
>table. I know that this has come up in the past, so you can probably
>find someone else who's already done this and might be able to provide
>you with code. Even if you have to write it from scratch, I doubt it's
>more work than what you've already put into trying to get the other
>scheme to work.
>
Thanks.Do you think that using LISTEN/NTIFY mechanism for rolling up
the summary table
could be used?

Sure, although on a busy system it might be better to just poll once a
minute or so. I'm not sure what the overhead for a NTIFY is, but I
imagine it would start to add up if you were calling several times a
second.