[core] Questions/Review on draft-ietf-core-resource-directory-12

[Jim Schaad]

Over all I think that I understand this version better, but there are still
some things that I am going to need to ask questions about or comment on.
Note this is not a pass from writing code, just from trying to do some
design work.


1. When creating a group registration, can the link to an endpoint be on a
different resource directory? This second RD could be either on a different
machine or on the same machine (i.e. coap vs coaps). If the answer is no,
then do we need to think about some type of relation attribute in the case
that multiple are listed somewhere?

2. Is there supposed to be an update/patch operation on groups or is this a
replace operation ala what is done for endpoints?

3. Are groups supposed to age out of existence? Based on the current text I
would say that this is a no.

4. Should have text on what happens to a group when an endpoint which ages
out of existence.

5. When registering groups, is there a requirement that all of the
endpoints be in the same domain as the group is?

6. In section 7.3 - First it would be better to be much more explicit on
the set of items to be matched than the one statement that is current
present. From what is there, it is not clear to me that one would return a
group if one of the resources of an endpoint in the group has the desired
attribute. However, based on everything else, it would appear to be true.
Thus the search would be to go up the model (Figure 2) to all ancestor notes
and down the tree to all descendent notes. Is this a correct understanding?

7. In section 7.3 - If doing a compound search criteria - Does all of the
criteria need to be matched on ANY node in the search or ALL of the criteria
on one node in the search? Consider "?d=example.com&rt=foobar" which would
match the 'd' on the endpoint but the 'rt' on a resource. So two questions
- the first would be looking at two different items at two different levels
(i.e. group and resource), the second would be two items at the same level
(i.e. two different resources/endpoints which are children/grandchildren of
me).


Jim

[Christian Amsüss]

Hello Jim,

Post by Jim Schaad
1. When creating a group registration, can the link to an endpoint be on a
different resource directory? This second RD could be either on a different
machine or on the same machine (i.e. coap vs coaps). If the answer is no,
then do we need to think about some type of relation attribute in the case
that multiple are listed somewhere?

I like to think so (having a more distributed model in mind), but we
haven't discussed it yet.

As for the specification, I think we should say that the CTs operating
on the tool working on the group "SHOULD refer to the endpoint
registrations either as a path-absolute relative reference (if it
obtained the reference from the Location option when it conducted the
registration itself), or use the reference as serialized when it
obtained the location from an endpoint lookup from the same RD origin".

This leaves the door open for federating RDs (where some endpoint
registrations are located at another RD host, but still available for
lookup at the closest host), but discourages blind mixing of
registration entities from unrelated RDs.

(Discussion about this can go continue on-list, but for tracking
purposes it's tracked at [85] too).

Post by Jim Schaad
2. Is there supposed to be an update/patch operation on groups or is this a
replace operation ala what is done for endpoints?

Yes. Whatever will work for the more intricate resource link update
should also work for the more set-like group management.

Right now, groups can only be replaced by re-registering with the same
group name and a new set of links.

Post by Jim Schaad
3. Are groups supposed to age out of existence? Based on the current text I
would say that this is a no.

So do I.

Post by Jim Schaad
4. Should have text on what happens to a group when an endpoint which ages
out of existence.

Good point, taken up ([86]).

Post by Jim Schaad
5. When registering groups, is there a requirement that all of the
endpoints be in the same domain as the group is?

That's the idea implied in the hierarchy, but probably answered better
by Michael or Peter who use the domain concept more than I do.

Post by Jim Schaad
6. In section 7.3 - First it would be better to be much more explicit on
the set of items to be matched than the one statement that is current
present. From what is there, it is not clear to me that one would return a
group if one of the resources of an endpoint in the group has the desired
attribute. However, based on everything else, it would appear to be true.
Thus the search would be to go up the model (Figure 2) to all ancestor notes
and down the tree to all descendent notes. Is this a correct understanding?

That is a good point and an open issue[82], and indeed the
`/rd-lookup/gp?ep=node1` example contradicts the current text that only
covers going "up" in Figure 2.

I'm leaning strongly toward allowing both directions, ie. a query
parameter matches a candidate result entity if it matches on its
parameters or the link parameters of any of its containers or its
contents, transitively.

What worries me a bit is the performance impact of looking for possibly
matching resource attributes in any group lookup, but probably it
shouldn't.

Post by Jim Schaad
7. In section 7.3 - If doing a compound search criteria - Does all of the
criteria need to be matched on ANY node in the search or ALL of the criteria
on one node in the search? Consider "?d=example.com&rt=foobar" which would
match the 'd' on the endpoint but the 'rt' on a resource. So two questions
- the first would be looking at two different items at two different levels
(i.e. group and resource), the second would be two items at the same level
(i.e. two different resources/endpoints which are children/grandchildren of
me).

The quantifier sequence is "for each query parameter, there exists a
(suitably related) entity that matches the parameter". Each query
parameter can be satisfied by a different entity.

I don't fully understand the part after 'Consider', but
`?d=exmaple.com&rt=foobar` could give a single endpoint in an EP lookup
(which is the only endpoint to contain rt=foobar resources), but give
both its two rt=foobar resources in a resource lookup.

A query `?rt=foobar&rt=fnord` lookup could match on the endpoint lookup
interface (finding an endpoint that contains both a foobar and a fnord),
but might not give any result on the resource lookup interface if no
single resource has both types.

I'll try to make that clearer when addressing the abovementioned issue
of going up and down -- but did that answer your question?


Thank you for your comments
Christian

[82]: https://github.com/core-wg/resource-directory/issues/82
[85]: https://github.com/core-wg/resource-directory/issues/85
[86]: https://github.com/core-wg/resource-directory/issues/86

[Jim Schaad]

-----Original Message-----
Sent: Thursday, November 2, 2017 2:27 AM
Subject: Re: Questions/Review on draft-ietf-core-resource-directory-12
Hello Jim,

Post by Jim Schaad
1. When creating a group registration, can the link to an endpoint be
on a different resource directory? This second RD could be either on
a different machine or on the same machine (i.e. coap vs coaps). If
the answer is no, then do we need to think about some type of relation
attribute in the case that multiple are listed somewhere?

I like to think so (having a more distributed model in mind), but we

haven't

discussed it yet.
As for the specification, I think we should say that the CTs operating on

the tool

working on the group "SHOULD refer to the endpoint registrations either as

a

path-absolute relative reference (if it obtained the reference from the

Location

option when it conducted the registration itself), or use the reference as
serialized when it obtained the location from an endpoint lookup from the
same RD origin".
This leaves the door open for federating RDs (where some endpoint
registrations are located at another RD host, but still available for

lookup at the

closest host), but discourages blind mixing of registration entities from
unrelated RDs.
(Discussion about this can go continue on-list, but for tracking purposes

it's

tracked at [85] too).

I really like the idea of being able to have entries in a group be on a
different server, right up to the point where I start trying to implement
the search algorithm that you have specified. At that point my code is
suddenly making lots of potential queries to the other RD in order to find
out if some parent/child is going to have the attribute that I am looking
for.

Post by Jim Schaad
4. Should have text on what happens to a group when an endpoint which
ages out of existence.

Good point, taken up ([86]).

Post by Jim Schaad
5. When registering groups, is there a requirement that all of the
endpoints be in the same domain as the group is?

That's the idea implied in the hierarchy, but probably answered better by
Michael or Peter who use the domain concept more than I do.

Post by Jim Schaad
6. In section 7.3 - First it would be better to be much more explicit
on the set of items to be matched than the one statement that is
current present. From what is there, it is not clear to me that one
would return a group if one of the resources of an endpoint in the
group has the desired attribute. However, based on everything else, it

would

appear to be true.

Post by Jim Schaad
Thus the search would be to go up the model (Figure 2) to all ancestor
notes and down the tree to all descendent notes. Is this a correct

understanding?
That is a good point and an open issue[82], and indeed the `/rd-
lookup/gp?ep=node1` example contradicts the current text that only covers
going "up" in Figure 2.
I'm leaning strongly toward allowing both directions, ie. a query

parameter

matches a candidate result entity if it matches on its parameters or the

link

parameters of any of its containers or its contents, transitively.
What worries me a bit is the performance impact of looking for possibly
matching resource attributes in any group lookup, but probably it

shouldn't.

It's interesting, I was sure there was an example along the lines of

GET /rd-lookup/ep?rt=temperature-c

But I don't see it there now.

Post by Jim Schaad
7. In section 7.3 - If doing a compound search criteria - Does all of
the criteria need to be matched on ANY node in the search or ALL of
the criteria on one node in the search? Consider
"?d=example.com&rt=foobar" which would match the 'd' on the endpoint
but the 'rt' on a resource. So two questions
- the first would be looking at two different items at two different
levels (i.e. group and resource), the second would be two items at the
same level (i.e. two different resources/endpoints which are
children/grandchildren of me).

The quantifier sequence is "for each query parameter, there exists a

(suitably

related) entity that matches the parameter". Each query parameter can be
satisfied by a different entity.
I don't fully understand the part after 'Consider', but
`?d=exmaple.com&rt=foobar` could give a single endpoint in an EP lookup
(which is the only endpoint to contain rt=foobar resources), but give both

its

two rt=foobar resources in a resource lookup.

It would be clearer if I said I was looking for a group or an end-point with
these parameters. I probably should have used something other than d= to
something that was clearly only set on a group.

A query `?rt=foobar&rt=fnord` lookup could match on the endpoint lookup
interface (finding an endpoint that contains both a foobar and a fnord),

but

might not give any result on the resource lookup interface if no single

resource

has both types.
I'll try to make that clearer when addressing the abovementioned issue of
going up and down -- but did that answer your question?

Yes, I think I understand the current desires.

Jim

Thank you for your comments
Christian
[82]: https://github.com/core-wg/resource-directory/issues/82
[85]: https://github.com/core-wg/resource-directory/issues/85
[86]: https://github.com/core-wg/resource-directory/issues/86
--
To use raw power is to make yourself infinitely vulnerable to greater powers.
-- Bene Gesserit axiom

['Christian Amsüss']

Post by Jim Schaad
I really like the idea of being able to have entries in a group be on a
different server, right up to the point where I start trying to implement
the search algorithm that you have specified. At that point my code is
suddenly making lots of potential queries to the other RD in order to find
out if some parent/child is going to have the attribute that I am looking
for.

I was envisioning a cluster where each node keeps the data registered
with the other ndoes in an eventually consistent cache; maybe I'll find
time to sketch that out a bit further before the meeting.

Post by Jim Schaad

Post by Christian Amsüss
What worries me a bit is the performance impact of looking for possibly
matching resource attributes in any group lookup, but probably it

shouldn't.
It's interesting, I was sure there was an example along the lines of
GET /rd-lookup/ep?rt=temperature-c
But I don't see it there now.

The example that's there is /rd-lookup/gp?ep=node1

Post by Jim Schaad

Post by Christian Amsüss

Consider "?d=example.com&rt=foobar" which would match the 'd' on
the endpoint but the 'rt' on a resource. So two questions - the
first would be looking at two different items at two different
levels (i.e. group and resource), the second would be two items at
the same level (i.e. two different resources/endpoints which are
children/grandchildren of me).

It would be clearer if I said I was looking for a group or an end-point with
these parameters. I probably should have used something other than d= to
something that was clearly only set on a group.

I see. Both these questions (`WHERE ep.d=="example.com"` and `JOIN ...
WHERE group.d=="example.com"`) are, in the simple query language of
search parameters, expressed in the same way and returned as their
union.

We don't have a partitioning on the names to tell the intended query
apart. (d can be on both endpoint and group, and there's no registry for
link parameters so we can't rule out conflicts).

What we could do to mitigate that ambiguity is to make the query
language slightly more complext (so your endpoint queries might be
`?d=example.com&res→rt=foobar` vs. `?gp→d=example.com&res→rt=foobar`),
but that won't reduce the complexity but only make it more visible.

Do you think that the availability of search in both directions of the
tree is an issue for implementations (esp. distributed one), or is it
more a problem of specification or interface clarity that the meanig of
the query is not immediately obvious?

Best regards
Christian

[peter van der Stok]

Hi Jim and Christian,

The discussion on a set of RDs is rather worrying to me.
I do understand the remark by Jim that the end-points composing a group
could be stored on different RDs than the RD handling the group
registration.
The RD is not aware of the remoteness of the eps and the client has to
handle the distribution.
It is not different from registering anchors to different hosts.

However, envisaging a set of RDs implies many protocols to keep entries
synchronized and handle communication errors.
I hope I misunderstood the intention of Christian.

Greetings,

Peter

Post by Christian Amsüss

Post by Christian Amsüss

Post by Jim Schaad
1. When creating a group registration, can the link to an endpoint be
on a different resource directory? This second RD could be either on
a different machine or on the same machine (i.e. coap vs coaps). If
the answer is no, then do we need to think about some type of relation
attribute in the case that multiple are listed somewhere?

I like to think so (having a more distributed model in mind), but we

haven't

Post by Christian Amsüss
discussed it yet.
As for the specification, I think we should say that the CTs operating on

the tool

Post by Christian Amsüss
working on the group "SHOULD refer to the endpoint registrations either as

a

Post by Christian Amsüss
path-absolute relative reference (if it obtained the reference from the

Location

Post by Christian Amsüss
option when it conducted the registration itself), or use the
reference as
serialized when it obtained the location from an endpoint lookup from the
same RD origin".
This leaves the door open for federating RDs (where some endpoint
registrations are located at another RD host, but still available for

lookup at the

Post by Christian Amsüss
closest host), but discourages blind mixing of registration entities from
unrelated RDs.
(Discussion about this can go continue on-list, but for tracking purposes

it's

Post by Christian Amsüss
tracked at [85] too).

I really like the idea of being able to have entries in a group be on a
different server, right up to the point where I start trying to implement
the search algorithm that you have specified. At that point my code is
suddenly making lots of potential queries to the other RD in order to find
out if some parent/child is going to have the attribute that I am looking
for.

[Jim Schaad]

Peter,

In the current design it is easy for the RD to understand the remoteness of
an included endpoint, it would have a full URL to a different RD rather than
having a partial URL which resolves against the current RD.

That said, I am not in favor of the idea because of the searching problems.
I would rather just have a specialized synchronization API in a different
document. (Which might be as easy as - I will publish any updates to the
following list of people.)

Jim

-----Original Message-----
Sent: Monday, November 6, 2017 12:41 AM
Subject: Re: [core] Questions/Review on

draft-ietf-core-resource-directory-12

Hi Jim and Christian,
The discussion on a set of RDs is rather worrying to me.
I do understand the remark by Jim that the end-points composing a group

could

be stored on different RDs than the RD handling the group registration.
The RD is not aware of the remoteness of the eps and the client has to

handle

the distribution.
It is not different from registering anchors to different hosts.
However, envisaging a set of RDs implies many protocols to keep entries
synchronized and handle communication errors.
I hope I misunderstood the intention of Christian.
Greetings,
Peter

Post by Christian Amsüss

Post by Christian Amsüss

Post by Jim Schaad
1. When creating a group registration, can the link to an endpoint
be on a different resource directory? This second RD could be
either on a different machine or on the same machine (i.e. coap vs
coaps). If the answer is no, then do we need to think about some
type of relation attribute in the case that multiple are listed

somewhere?

Post by Christian Amsüss

Post by Christian Amsüss
I like to think so (having a more distributed model in mind), but we

haven't

Post by Christian Amsüss
discussed it yet.
As for the specification, I think we should say that the CTs operating on

the tool

Post by Christian Amsüss
working on the group "SHOULD refer to the endpoint registrations either as

a

Post by Christian Amsüss
path-absolute relative reference (if it obtained the reference from the

Location

Post by Christian Amsüss
option when it conducted the registration itself), or use the
reference as serialized when it obtained the location from an
endpoint lookup from the same RD origin".
This leaves the door open for federating RDs (where some endpoint
registrations are located at another RD host, but still available for

lookup at the

Post by Christian Amsüss
closest host), but discourages blind mixing of registration entities
from unrelated RDs.
(Discussion about this can go continue on-list, but for tracking purposes

it's

Post by Christian Amsüss
tracked at [85] too).

I really like the idea of being able to have entries in a group be on
a different server, right up to the point where I start trying to
implement the search algorithm that you have specified. At that point
my code is suddenly making lots of potential queries to the other RD
in order to find out if some parent/child is going to have the
attribute that I am looking for.

[peter van der Stok]

Post by Christian Amsüss

Post by Jim Schaad
5. When registering groups, is there a requirement that all of the
endpoints be in the same domain as the group is?

That's the idea implied in the hierarchy, but probably answered better
by Michael or Peter who use the domain concept more than I do.

Sorry, a bit late in answering, for good reasons though.

My cut feeling is "no". Groups may be part of other domains than the
contributing endpoints.
One might think of domain hierarchies.
e.g. endpoint in room domains, multicast in enveloping corridor domain.
I prefer to leave this to users of RD and not add extra rules.

Peter

[Christian M. Amsüss]

Hello Peter, hello Jim,

Post by peter van der Stok
The discussion on a set of RDs is rather worrying to me.

First, please see this whole RD federation topic as a matter of dancing
on thin ice to see whether it tolds. IMO, if RDs can conceptually be
federated, this shows that the whole linking thing works out well, but
if not, we should not change the specification to make it work (unless
we discover true inconsistencies from that thougth experiment).

This is nothing to be expected to work out-of-the box; just something
that happens to be easily doable because of how awesome RD is these days ;-)


I've added the example of what this could like (promised in an earlier
mail) below.

Post by peter van der Stok
I would rather just have a specialized synchronization API in a different
document. (Which might be as easy as - I will publish any updates to the
following list of people.)

Do you think that'll benefit from being specified at all? The RD will be
an interaction point in heterogenous environments, but I doubt that
there will be much use for fully federated heterogenous RDs for some
time.


Best regards
Christian


---

A story of how federated RDs could work
=======================================

Say X and Y are federated RDs, Z is also federated but won't accept
registrations on its own. A, B and C are clients in close proxity to the
respective servers.

All clients behave only as per RD specification, with some tolerance to
non-local references during discovery.

Say C wants to register, and SLAAC gave it the address of Z:

| GET coap://Z/.well-known/core?rt=core.rd
| 2.05 Content
| <coap://Y/rd>;rt="core.rd"

(because Z is just doing replication)

| POST coap://Y/rd?ep=node-C
| </temp>;if="core.s";rt="temperature"
| 2.04 Created
| Location: /reg/1234

Since the beginning of federation, X and Z have been observing endpoints
and resources, and now get an update. The `only_local` thing is an RD
parameter registered for RD federation to prevents loops, and I'm
assuming a PATCH-style observation (even though it's not there yet):

| GET coap://Y/lookup/ep?only_local, Observe
| 2.06 Partial Content
| New record </reg/1234>;ep="node-C";lt=84600

and on their second observation:

| GET coap://Y/lookup/res?only_local, Observe
| 2.06 Partial Content
| New record </temp>;if="core.s";rt="temperature";anchor="coap://C"

If now any client asks X, it can get

| GET coap://X/lookup/res?rt=temperature
| 2.05 Content
| </temp>;if="core.s";rt="temperature";anchor="coap://C"

without noticing anything, or it could look for all registered nodes
with core.s interfaces and get some local and some remote ones:

| GET coap://X/lookup/ep?if=core.s
| 2.05 Content
| </reg/991>;ep="node-A",
| </reg/1200>;ep="node-B",;anchor="coap://Y",
| </reg/1234>;ep="node-C";anchor="coap://Y"

When RD Y prepares to shut down, it'll convey its federation membership
status of "retreating" to the other members (however tha's done), so Z
will no more announce its directory resource but the next closest one
(X). Later in that process, when C renews its membership, Y will cancel
it and respond 4.04, triggering C to retry the registration process,
find X, and get a new registrtion resource there. When all of Y's active
registrations have migrated or timed out, Y can quit the federation.

(Some RD endpoint implementations (shameless plug: aiocoap's) might be
robust enough to even tolerate Y silently going down if Z announced

| GET coap://Z/.well-known/core?rt=core.rd
| 2.05 Content
| <coap://Y/rd>;rt="core.rd",
| <coap://X/rd>;rt="core.rd"

; that won't do no harm, but as we don't specify any round-robin there,
it won't replace active monitoring of federated RDs).