LDAP group query optimisation

[Brian Candler]

I am testing out freeradius with FreeIPA (= 389 directory server). This
is freeradius-3.0.11 from Ubuntu 16.04, talking to FreeIPA under CentOS 7.

The 389 directory server in FreeIPA has a "memberOf" plugin installed
(by default), which exposes all the groups as part of the user record.
For example:

# bcandler, users, accounts, ipa.example.com
dn: uid=bcandler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com
objectclass: ipaobject
objectclass: person
objectclass: top
objectclass: ipasshuser
objectclass: inetorgperson
objectclass: organizationalperson
objectclass: krbticketpolicyaux
objectclass: krbprincipalaux
objectclass: inetuser
objectclass: posixaccount
objectclass: ipaSshGroupOfPubKeys
objectclass: mepOriginEntry
objectclass: ipantuserattrs
cn: Brian Candler
memberOf: cn=ipausers,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com
memberOf: cn=server_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com
memberOf: cn=network_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com
memberOf: cn=vpn,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com
memberOf: cn=staff,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com

And I have freeradius's ldap module configured for group membership like
this:

group {
membership_attribute = 'memberOf'
name_attributes = 'cn'
}


The problem is, whenever I touch the LDAP-Group attribute it triggers
off a whole load of LDAP queries, one for each group, to translate the
group DN to the cn.

However, since all I'm asking for the cn, and the cn is the RDN of the
group, the cn could be extracted directly from the DN.

Here's an example of what I see:


(1) if (&Called-Station-Id =~ /:Staff$/ && &LDAP-Group[*] ==
"staff") {
(1) Searching for user in group "staff"
rlm_ldap (ldap): Reserved connection (3)
(1) Using user DN from request
"uid=bcandler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com"
(1) Checking user object's memberOf attributes
(1) Performing unfiltered search in
"uid=bcandler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com", scope "base"
(1) Waiting for search result...
(1) Processing memberOf value
"cn=ipausers,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" as a DN
(1) Resolving group DN
"cn=ipausers,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" to group name
(1) Performing unfiltered search in
"cn=ipausers,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com", scope "base"
(1) Waiting for search result...
(1) Group DN
"cn=ipausers,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" resolves to
name "ipausers"
(1) Processing memberOf value
"cn=server_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" as a DN
(1) Resolving group DN
"cn=server_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" to group
name
(1) Performing unfiltered search in
"cn=server_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com", scope
"base"
(1) Waiting for search result...
(1) Group DN
"cn=server_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" resolves
to name "server_guru"
(1) Processing memberOf value
"cn=network_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" as a DN
(1) Resolving group DN
"cn=network_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" to
group name
(1) Performing unfiltered search in
"cn=network_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com", scope
"base"
(1) Waiting for search result...
(1) Group DN
"cn=network_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com"
resolves to name "network_guru"
(1) Processing memberOf value
"cn=vpn,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" as a DN
(1) Resolving group DN
"cn=vpn,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" to group name
(1) Performing unfiltered search in
"cn=vpn,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com", scope "base"
(1) Waiting for search result...
(1) Group DN
"cn=vpn,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" resolves to name
"vpn"
(1) Processing memberOf value
"cn=staff,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" as a DN
(1) Resolving group DN
"cn=staff,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" to group name
(1) Performing unfiltered search in
"cn=staff,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com", scope "base"
(1) Waiting for search result...
(1) Group DN
"cn=staff,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com" resolves to
name "staff"
(1) User found in group "staff". Comparison between membership:
name (resolved from DN
"cn=staff,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com"), check: name


I guess this is intentional: one group object *could* have multiple cn
attributes, so maybe it's querying the group to be sure. That is,

memberOf: cn=ipausers,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com

in theory could translate to

LDAP-Group += ipausers
LDAP-Group += another name for ipausers


However in my case I don't need this. Is there a way I can configure
the LDAP module not to do this?

Alternatively I could make an explicit group membership query (i.e.
return all group entries that this user is a member of); but that still
involves two queries, and I would then not be making use of the memberOf
feature.

Thanks,

Brian.

-
List info/subscribe/unsubscr

[Alan DeKok]

I am testing out freeradius with FreeIPA (= 389 directory server). This is freeradius-3.0.11 from Ubuntu 16.04, talking to FreeIPA under CentOS 7.

...

The problem is, whenever I touch the LDAP-Group attribute it triggers off a whole load of LDAP queries, one for each group, to translate the group DN to the cn.

Try the v3.0.x branch. It's largely v3, with a number of changes. One major one is LDAP group caching.

The LDAP module grabs all of the groups once, and then caches them along with the request. Subsequent LDAP group comparisons are done internally, and don't touch LDAP.

Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/

[Arran Cudbard-Bell]

I am testing out freeradius with FreeIPA (= 389 directory server). This is freeradius-3.0.11 from Ubuntu 16.04, talking to FreeIPA under CentOS 7.

...

The problem is, whenever I touch the LDAP-Group attribute it triggers off a whole load of LDAP queries, one for each group, to translate the group DN to the cn.

Specify the group as a DN and it won't do the translation.

If you want to examine the DN values yourself using a foreach loop, toggle on cacheable_dn, all the membership DNs will then be available as LDAP-Group instances.

I can see some advantages do extracting groups from the RDN, so feel free to add an issue ticket.

-Arran

[Brian Candler]

Post by Alan DeKok
Try the v3.0.x branch. It's largely v3

I don't get that. Did you mean "Try the v4.0.x branch" ?


-
List info/subscribe/unsu

[Alan Buxey]

No. I think he did mean 3.0.x - ie the latest version of 3.0.x via GIT ... that'll be 3.0.13 when released, its currently 3.0.12 with some extra bugs/issues fixed (read release notes and/or git submission logs )

alan
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/

[Brian Candler]

No. I think he did mean 3.0.x - ie the latest version of 3.0.x via GIT
... that'll be 3.0.13 when released, its currently 3.0.12 with some
extra bugs/issues fixed (read release notes and/or git submission logs )

I did look at "git diff release_3_0_11 head" before posting.

I saw nothing in doc/Changelog relating to LDAP between 3.0.11 and
3.0.13(pre), except

* Don't complain on /dev/urandom in ldap

And I saw some small tweaks to src/modules/rlm_ldap/rlm_ldap.c, but
didn't note anything which would change the fundamental logic.

B.

-
List info/subscribe/unsubsc

[A***@lboro.ac.uk]

Hi,

Post by Brian Candler

Post by Alan DeKok
Try the v3.0.x branch. It's largely v3

I don't get that. Did you mean "Try the v4.0.x branch" ?

maybe he did mean 4.0.x - which is interesting as previously told to
stay clear of it on production systems especially as its about to go all config change crazy.

however, if there is now some change to that policy and 4.0.x is to be very much
3.x with tweaks - with the new crazy stuff being 4.1.x or such then please let
us know Alan :/


alan
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/l

[Alan DeKok]

Post by A***@lboro.ac.uk
Hi,

Post by Brian Candler

Post by Alan DeKok
Try the v3.0.x branch. It's largely v3

I don't get that. Did you mean "Try the v4.0.x branch" ?

maybe he did mean 4.0.x - which is interesting as previously told to
stay clear of it on production systems especially as its about to go all config change crazy.

I meant v3.1.x, sorry.

Post by A***@lboro.ac.uk
however, if there is now some change to that policy and 4.0.x is to be very much
3.x with tweaks - with the new crazy stuff being 4.1.x or such then please let
us know Alan :/

I'm going to be dropping much of 4.0 on the floor in the next week. For a while, it will likely be only a UDP server. No TLS or TCP. Then once the re-architecture is fixed, TCP and TLS become much easier.

Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.htm

[Brian Candler]

Post by Arran Cudbard-Bell
Specify the group as a DN and it won't do the translation.

Thanks, that works.

I guess there is some implicit logic which checks to see if the string
tastes like a DN or not. Ah yes: rlm_ldap_is_dn().

Post by Arran Cudbard-Bell
If you want to examine the DN values yourself using a foreach loop,

toggle on cacheable_dn, all the membership DNs will then be available as
LDAP-Group instances.

OK, that works too. It materializes LDAP-Group and LDAP-Group[*], which
can then also be used in string expansions.

I didn't investigate this flag before, because it said it was for use
with rlm_cache, which I'm not using.

Some documentation on the magical behaviour of the LDAP-Group attribute
would be nice to have :-)

Post by Arran Cudbard-Bell
I can see some advantages do extracting groups from the RDN, so feel

free to add an issue ticket.

Done: https://github.com/FreeRADIUS/freeradius-server/issues/1788

Cheers,

Brian.
-
List info/subscribe/u

[Arran Cudbard-Bell]

Post by Brian Candler

Post by Arran Cudbard-Bell
Specify the group as a DN and it won't do the translation.

Thanks, that works.
I guess there is some implicit logic which checks to see if the string tastes like a DN or not. Ah yes: rlm_ldap_is_dn().

:)

Post by Brian Candler

Post by Arran Cudbard-Bell
If you want to examine the DN values yourself using a foreach loop, toggle on cacheable_dn, all the membership DNs will then be available as LDAP-Group instances.

OK, that works too. It materializes LDAP-Group and LDAP-Group[*], which can then also be used in string expansions.

Well it converts the groups to VP instances, LDAP-Group[*] is just the all instance selector.

Post by Brian Candler
I didn't investigate this flag before, because it said it was for use with rlm_cache, which I'm not using.
Some documentation on the magical behaviour of the LDAP-Group attribute would be nice to have :-)

RTFC :p

Actually RTFMing reveals:

# If cacheable_name or cacheable_dn are enabled,
# all group information for the user will be
# retrieved from the directory and written to LDAP-Group
# attributes appropriate for the instance of rlm_ldap.

I added a note about it not just being for rlm_cache.

Post by Brian Candler

Post by Arran Cudbard-Bell
I can see some advantages do extracting groups from the RDN, so feel free to add an issue ticket.

Done: https://github.com/FreeRADIUS/freeradius-server/issues/1788

Thanks.

-Arran

[Brian Candler]

Post by Arran Cudbard-Bell
# If cacheable_name or cacheable_dn are enabled,
# all group information for the user will be
# retrieved from the directory and written to LDAP-Group
# attributes appropriate for the instance of rlm_ldap.
I added a note about it not just being for rlm_cache.

Great. That text did mean anything without the knowledge that
control:LDAP-Group is not a "real" attribute, unless you turn on those
cacheable_xxxx attributes.

Instead, it is a magical pseudo attribute which triggers
behind-the-scenes queries when you match on it.

Now grepping for paircompare_register in the code, it looks like there
are some other attributes which might have similar magical powers.

-
List info/subscribe/unsu

[Brian Candler]

A couple of other points about LDAP-Group.

(1) I tried adding the following for debugging inside a policy module
(invoked after the LDAP lookup has taken place):

update control {
Tmp-String-0 := "%{request:LDAP-Group}"
Tmp-String-1 := "%{control:LDAP-Group}"
Tmp-String-2 := "%{reply:LDAP-Group}"
Tmp-String-3 := "%{request:LDAP-Group[*]}"
Tmp-String-4 := "%{control:LDAP-Group[*]}"
Tmp-String-5 := "%{reply:LDAP-Group[*]}"
}

All six variations expand to empty:

rlm_ldap (ldap): Bind successful
(0) [ldap] = updated
(0) policy group_authorization {
(0) update control {
(0) EXPAND %{request:LDAP-Group}
(0) -->
(0) Tmp-String-0 :=
(0) EXPAND %{control:LDAP-Group}
(0) -->
(0) Tmp-String-1 :=
(0) EXPAND %{reply:LDAP-Group}
(0) -->
(0) Tmp-String-2 :=
(0) EXPAND %{request:LDAP-Group[*]}
(0) -->
(0) Tmp-String-3 :=
(0) EXPAND %{control:LDAP-Group[*]}
(0) -->
(0) Tmp-String-4 :=
(0) EXPAND %{reply:LDAP-Group[*]}
(0) -->
(0) Tmp-String-5 :=
(0) } # update control = noop

Changing %{...} to %{&...} didn't make any difference either.

Is this expected behaviour? If the LDAP-Group pseudo-attribute is not
valid in string expansions, perhaps it should at least generate a warning?


(2) I found another fragility issue with LDAP-Group: if you are
comparing this attribute but miss the leading &, it silently fails,
without even so much as a warning.

Here is a snippet from a working policy:

if (&Huntgroup-Name == "wifi") {
if (&Called-Station-Id =~ /:Admin$/ && ( &LDAP-Group[*] ==
"server_guru" || &LDAP-Group[*] == "network_guru")) {
return
}

Debug is happy:

(0) if (&Huntgroup-Name == "wifi") {
(0) if (&Huntgroup-Name == "wifi") -> TRUE
(0) if (&Huntgroup-Name == "wifi") {
(0) if (&Called-Station-Id =~ /:Admin$/ && ( &LDAP-Group[*] ==
"server_guru" || &LDAP-Group[*] == "network_guru")) {
(0) Searching for user in group "server_guru"
rlm_ldap (ldap): Reserved connection (1)
(0) Using user DN from request
"uid=bcandler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com"
(0) Checking user object's memberOf attributes
... snip
(0) User found in group "server_guru". Comparison between
membership: name (resolved from DN
"cn=server_guru,cn=groups,cn=accounts,dc=ipa,dc=example,dc=com"), check:
name
rlm_ldap (ldap): Released connection (1)
(0) if (&Called-Station-Id =~ /:example Admin$/ && (
&LDAP-Group[*] == "server_guru" || &LDAP-Group[*] == "network_guru"))
-> TRUE

However in the first iteration of writing this policy, I had
accidentally missed out one of the & signs. To reproduce the issue, I
have changed it to this:

if (&Huntgroup-Name == "wifi") {
if (&Called-Station-Id =~ /:Admin$/ && ( LDAP-Group[*] ==
"server_guru" || LDAP-Group[*] == "network_guru")) {
return
}

The only difference is dropping & before the two instances of LDAP-Group[*]

The debug output now looks like this:

(0) if (&Huntgroup-Name == "wifi") {
(0) if (&Called-Station-Id =~ /:example Admin$/ && (
LDAP-Group[*] == "server_guru" || LDAP-Group[*] == "network_guru")) {
(0) if (&Called-Station-Id =~ /:example Admin$/ && (
LDAP-Group[*] == "server_guru" || LDAP-Group[*] == "network_guru")) ->
FALSE

That's it. The whole expression fails, and there's no other indication
that I did something wrong.

This, I think, could lead to some difficult-to-find bugs in policy. And
the unlang documentation does say that the & is optional for attributes
on the left-hand side of a comparison operator:

" Where the left-hand side is an attribute, the "&" can be omitted."

Regards,

Brian.
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.