Simo,
A while ago, we discussed my desire for the paged results control to be
enabled by default, on all LDAP queries.
For the benefit of the list, I was discussing with simo a problem I was
having, with downloading a large amount of data from AD's LDAP server.
At the current time, I'm looking to download schema information from
AD's LDAP server, but I'm likewise interested in how an async winbindd
could be built, given we now have the infrastructure.
I know I can specify magic options to ldbsearch, but this misses the
point:
My assertion is that the caller, and in the case of the ldbsearch
binary, the user, should have this 'just work'. If the search would
return too many entries for one try, it should retry with paged results
transparently.
I'm happy to make it possible for the caller to disable it, but I would
like the control to be on by default, as otherwise there is a differece
in behaviour between the TDB and LDAP backends.
This would also allow our clients to make large searches via our LDAP
server, where we can then apply the sort control and similar.
, clients can only do large searches against tdb (which has no
such size limit).
Andrew Bartlett

Mon, 2006-08-14 at 11:43 +1000, Andrew Bartlett wrote:
Simo,

A while ago, we discussed my desire for the paged results control to be
enabled by default, on all LDAP queries.

For the benefit of the list, I was discussing with simo a problem I was
having, with downloading a large amount of data from AD's LDAP server.

At the current time, I'm looking to download schema information from
AD's LDAP server, but I'm likewise interested in how an async winbindd
could be built, given we now have the infrastructure.

I know I can specify magic options to ldbsearch, but this misses the
point:

My assertion is that the caller, and in the case of the ldbsearch
binary, the user, should have this 'just work'. If the search would
return too many entries for one try, it should retry with paged results
transparently.

I'm happy to make it possible for the caller to disable it, but I would
like the control to be on by default, as otherwise there is a differece
in behaviour between the TDB and LDAP backends.

This would also allow our clients to make large searches via our LDAP
server, where we can then apply the sort control and similar.
, clients can only do large searches against tdb (which has no
such size limit).

We can do this, but I want it done right.

The problem with the paged control is that it requires the caller to
make multiple calls.
Also we cannot use the control when it is not supported by the remote
server.

So the right way to do it is to do it inside ldb_search(), and to not
repeat a rootDSE search at every ldb_search() we need to keep the
information about whether the server supports paged results or not
somewhere. In theory the right place would be the ldb backend, but then
we would need a way to query it to know whether or not we are allowed to
use the paged search control.

What we can't do, is to query the rootDSE at each search, or forcibly
add the paged control at any serach in the ldb backend, because when you
use the ldb_request(0 call it means you want complete control over what
happens on the wire, you don't want "surprises".

So provided we find a decent method to make ldb_search() know when to
use the paged results control I am ok adding the code to do it there
automatically.

Simo.

Mon, 2006-08-14 at 01:49 -0400, simo wrote:
Mon, 2006-08-14 at 11:43 +1000, Andrew Bartlett wrote:
Simo,

A while ago, we discussed my desire for the paged results control to be
enabled by default, on all LDAP queries.

For the benefit of the list, I was discussing with simo a problem I was
having, with downloading a large amount of data from AD's LDAP server.

At the current time, I'm looking to download schema information from
AD's LDAP server, but I'm likewise interested in how an async winbindd
could be built, given we now have the infrastructure.

I know I can specify magic options to ldbsearch, but this misses the
point:

My assertion is that the caller, and in the case of the ldbsearch
binary, the user, should have this 'just work'. If the search would
return too many entries for one try, it should retry with paged results
transparently.

I'm happy to make it possible for the caller to disable it, but I would
like the control to be on by default, as otherwise there is a differece
in behaviour between the TDB and LDAP backends.

This would also allow our clients to make large searches via our LDAP
server, where we can then apply the sort control and similar.
, clients can only do large searches against tdb (which has no
such size limit).

We can do this, but I want it done right.

The problem with the paged control is that it requires the caller to
make multiple calls.

Why can't the backend re-issue the request for the remaining attributes?
That way, the caller only does one call, and the async code works
exactly how we want: it returns the results when available.

Also we cannot use the control when it is not supported by the remote
server.

Indeed. But I presume we don't need to mark it critical?

So the right way to do it is to do it inside ldb_search(), and to not
repeat a rootDSE search at every ldb_search() we need to keep the
information about whether the server supports paged results or not
somewhere. In theory the right place would be the ldb backend, but then
we would need a way to query it to know whether or not we are allowed to
use the paged search control.

The ldb_ildap backend already does a rootDSE search at startup, to find
out other things. Looking for that control would not be particularly
painful.

What we can't do, is to query the rootDSE at each search, or forcibly
add the paged control at any serach in the ldb backend, because when you
use the ldb_request(0 call it means you want complete control over what
happens on the wire, you don't want "surprises".

I'm happy to have a 'no supprises' flag, to turn this off, or an
'automagic' flag to turn this on, or a module that sets this, or
something. But it would be pointless if the ldb_request interface,
which is what an async winbind would use doesn't do the 'right thing' by
default.

So provided we find a decent method to make ldb_search() know when to
use the paged results control I am ok adding the code to do it there
automatically.

I really think this belongs in the ldb_ildap backend. But talk without
code is pointless, so I'll try and code something up.

Andrew Bartlett

Mon, 2006-08-14 at 16:37 +1000, Andrew Bartlett wrote:
Mon, 2006-08-14 at 01:49 -0400, simo wrote:

We can do this, but I want it done right.

The problem with the paged control is that it requires the caller to
make multiple calls.

Why can't the backend re-issue the request for the remaining attributes?
That way, the caller only does one call, and the async code works
exactly how we want: it returns the results when available.

Because it will make the module much more complex, and will add the kind
of smartness that lead to surprises later on when you want to control
closely what the backend is doing.

Also we cannot use the control when it is not supported by the remote
server.

Indeed. But I presume we don't need to mark it critical?

I think it is better to check beforehand so that you now if you have to
enter a call loop to munge the data, or use the simple way.

So the right way to do it is to do it inside ldb_search(), and to not
repeat a rootDSE search at every ldb_search() we need to keep the
information about whether the server supports paged results or not
somewhere. In theory the right place would be the ldb backend, but then
we would need a way to query it to know whether or not we are allowed to
use the paged search control.

The ldb_ildap backend already does a rootDSE search at startup, to find
out other things. Looking for that control would not be particularly
painful.

I know, but this is not a good reason to push for this stuff to be put
in there.

What we can't do, is to query the rootDSE at each search, or forcibly
add the paged control at any serach in the ldb backend, because when you
use the ldb_request(0 call it means you want complete control over what
happens on the wire, you don't want "surprises".

I'm happy to have a 'no supprises' flag, to turn this off, or an
'automagic' flag to turn this on, or a module that sets this, or
something. But it would be pointless if the ldb_request interface,
which is what an async winbind would use doesn't do the 'right thing' by
default.

, I think this is a perfect match for our first "client side" module.
I can make it up if you want.

So provided we find a decent method to make ldb_search() know when to
use the paged results control I am ok adding the code to do it there
automatically.

I really think this belongs in the ldb_ildap backend. But talk without
code is pointless, so I'll try and code something up.

If you can do it as a module to be loaded just before ldb_ildap, I think
we will get the best way to address this problem.

Simo.

Mon, 2006-08-14 at 12:20 -0400, simo wrote:

I'm happy to have a 'no supprises' flag, to turn this off, or an
'automagic' flag to turn this on, or a module that sets this, or
something. But it would be pointless if the ldb_request interface,
which is what an async winbind would use doesn't do the 'right thing' by
default.

, I think this is a perfect match for our first "client side" module.
I can make it up if you want.

That would be great.

So provided we find a decent method to make ldb_search() know when to
use the paged results control I am ok adding the code to do it there
automatically.

I really think this belongs in the ldb_ildap backend. But talk without
code is pointless, so I'll try and code something up.

If you can do it as a module to be loaded just before ldb_ildap, I think
we will get the best way to address this problem.

I agree.

Thanks,

Andrew Bartlett

Tue, 2006-08-15 at 07:46 +1000, Andrew Bartlett wrote:
Mon, 2006-08-14 at 12:20 -0400, simo wrote:

I'm happy to have a 'no supprises' flag, to turn this off, or an
'automagic' flag to turn this on, or a module that sets this, or
something. But it would be pointless if the ldb_request interface,
which is what an async winbind would use doesn't do the 'right thing' by
default.

, I think this is a perfect match for our first "client side" module.
I can make it up if you want.

That would be great.

Now that we have this module, what would be a good way to ensure that
client tools use it, when talking to an LDAP server?

That would avoid needing to remember to add the right magic to the
ldbsearch/ad2oLschema etc command line.

Andrew Bartlett

Fri, 2006-09-01 at 10:59 +1000, Andrew Bartlett wrote:
Tue, 2006-08-15 at 07:46 +1000, Andrew Bartlett wrote:
Mon, 2006-08-14 at 12:20 -0400, simo wrote:

I'm happy to have a 'no supprises' flag, to turn this off, or an
'automagic' flag to turn this on, or a module that sets this, or
something. But it would be pointless if the ldb_request interface,
which is what an async winbind would use doesn't do the 'right thing' by
default.

, I think this is a perfect match for our first "client side" module.
I can make it up if you want.

That would be great.

Now that we have this module, what would be a good way to ensure that
client tools use it, when talking to an LDAP server?

That would avoid needing to remember to add the right magic to the
ldbsearch/ad2oLschema etc command line.

Actually the best way is to add an options string manually in the code,
checking if the user has not already set any module options.
We may need to think of a better way to specify modules from inside the
program.

Simo.