[core] TCP, UDP, Proxy and Observe

[Jim Schaad]

I have been sketching out what my TCP/UDP and UDP/TCP proxy code would look
like and I am having a problem deciding what should be done for the observe
option.

If you have the case of Client->UDP->Proxy->UDP->Server, then dealing with
out of order observations can be dealt with in the following ways:

1. Pass them through unchanged or subtract a constant from the observer
value. This pushes the problem of out of order observes onto the client.
The proxy would save the high water content.
2. Filter on observe value so that out of order transmissions are
suppressed.
3. Keep out of order items around for a while and see if the out of order
items show up and transmit them back as in order. I missing items do not
show up until the time out, then suppress the out of order items.
4. Reorder the numbers so that things are seen in the wrong order by the
client. (Obviously a wrong answer.)

I assumed that 1 was the correct answer and deal with things that way.

Now I come to the question of Client->TCP->Proxy->UDP->Server. Since the
observe value is now to be ignored or removed, the first option above will
no longer work and I am not sure which the remaining options is the correct
one to do. Does a reliable transmission protocol mean that we should
suppress items?

Jim

[Robert Quattlebaum]

Post by Jim Schaad
Now I come to the question of Client->TCP->Proxy->UDP->Server. Since the
observe value is now to be ignored or removed, the first option above will
no longer work and I am not sure which the remaining options is the correct
one to do. Does a reliable transmission protocol mean that we should
suppress items?

According to the data model that observe is based on, out-of-order observe response packets are considered to be out-of-date and thus should be ignored. Proxies must be aware of outstanding observations and keep track of the observe counters to make sure that out-of-order packets are dropped. I'm not super happy about the mechanism, but that seems to be what the standard has settled on. Passing everything through and letting the client sort it out is explicitly not allowed.

-- RQ

Post by Jim Schaad
I have been sketching out what my TCP/UDP and UDP/TCP proxy code would look
like and I am having a problem deciding what should be done for the observe
option.
If you have the case of Client->UDP->Proxy->UDP->Server, then dealing with
1. Pass them through unchanged or subtract a constant from the observer
value. This pushes the problem of out of order observes onto the client.
The proxy would save the high water content.
2. Filter on observe value so that out of order transmissions are
suppressed.
3. Keep out of order items around for a while and see if the out of order
items show up and transmit them back as in order. I missing items do not
show up until the time out, then suppress the out of order items.
4. Reorder the numbers so that things are seen in the wrong order by the
client. (Obviously a wrong answer.)
I assumed that 1 was the correct answer and deal with things that way.
Now I come to the question of Client->TCP->Proxy->UDP->Server. Since the
observe value is now to be ignored or removed, the first option above will
no longer work and I am not sure which the remaining options is the correct
one to do. Does a reliable transmission protocol mean that we should
suppress items?
Jim
_______________________________________________
core mailing list
https://www.ietf.org/mailman/listinfo/core

[Carsten Bormann]

Post by Robert Quattlebaum
According to the data model that observe is based on, out-of-order observe response packets are considered to be out-of-date and thus should be ignored.

Right, Observe gives you eventual consistency, where data that is not fresh is not of interest.

Grüße, Carsten

[Jim Schaad]

-----Original Message-----
From: Robert Quattlebaum [mailto:***@gmail.com]
Sent: Monday, April 24, 2017 9:46 AM
To: Jim Schaad <***@augustcellars.com>
Cc: core <***@ietf.org>
Subject: Re: [core] TCP, UDP, Proxy and Observe

Post by Jim Schaad
Now I come to the question of Client->TCP->Proxy->UDP->Server. Since
the observe value is now to be ignored or removed, the first option
above will no longer work and I am not sure which the remaining
options is the correct one to do. Does a reliable transmission
protocol mean that we should suppress items?

According to the data model that observe is based on, out-of-order observe
response packets are considered to be out-of-date and thus should be
ignored. Proxies must be aware of outstanding observations and keep track of
the observe counters to make sure that out-of-order packets are dropped. I'm
not super happy about the mechanism, but that seems to be what the standard
has settled on. Passing everything through and letting the client sort it
out is explicitly not allowed.

[JLS] I cannot quite see that as being a true statement. It is true that
the most reason transmission must be considered the freshest, however a
client may want a more complete history of values than just the most recent
one and can deal with a non-freshest value. For example, if it is a
controller and is doing projections based on the current set of values,
having the "stale" value is still useful information even if it is not the
freshest. I do not see anything in RFC 7641 that says that out of order
packets are dropped by an intermediary. Are you saying this just based on
the fact that it is not freshest?

Jim


-- RQ

Post by Jim Schaad
I have been sketching out what my TCP/UDP and UDP/TCP proxy code would
look like and I am having a problem deciding what should be done for
the observe option.
If you have the case of Client->UDP->Proxy->UDP->Server, then dealing
1. Pass them through unchanged or subtract a constant from the
observer value. This pushes the problem of out of order observes onto the

client.

Post by Jim Schaad
The proxy would save the high water content.
2. Filter on observe value so that out of order transmissions are
suppressed.
3. Keep out of order items around for a while and see if the out of
order items show up and transmit them back as in order. I missing
items do not show up until the time out, then suppress the out of order

items.

Post by Jim Schaad
4. Reorder the numbers so that things are seen in the wrong order by
the client. (Obviously a wrong answer.)
I assumed that 1 was the correct answer and deal with things that way.
Now I come to the question of Client->TCP->Proxy->UDP->Server. Since
the observe value is now to be ignored or removed, the first option
above will no longer work and I am not sure which the remaining
options is the correct one to do. Does a reliable transmission
protocol mean that we should suppress items?
Jim
_______________________________________________
core mailing list
https://www.ietf.org/mailman/listinfo/core

[Carsten Bormann]

Post by Jim Schaad
I do not see anything in RFC 7641 that says that out of order
packets are dropped by an intermediary. Are you saying this just based on
the fact that it is not freshest?

While there may be many kinds of proxy, the standard caching proxy will only put fresh data into the cache, and only those would lead to notifications.

Observing a resource may not be the best model if you want to keep an uninterrupted time series of the value of the resource.

Grüße, Carsten

[Robert Quattlebaum]

Even a non-caching proxy will have to drop out-of-order observations if it proxies across UDP and TLS. There is pretty much only one correct behavior in that case, considering that there is no way to preserve the observe counter across a draft-ietf-core-coap-tcp-tls-07-style stream.

Post by Carsten Bormann

Post by Jim Schaad
I do not see anything in RFC 7641 that says that out of order
packets are dropped by an intermediary. Are you saying this just based on
the fact that it is not freshest?

While there may be many kinds of proxy, the standard caching proxy will only put fresh data into the cache, and only those would lead to notifications.
Observing a resource may not be the best model if you want to keep an uninterrupted time series of the value of the resource.
GrÌße, Carsten

[Robert Quattlebaum]

Hello Jim,

Post by Jim Schaad

Post by Robert Quattlebaum
According to the data model that observe is based on, out-of-order observe
response packets are considered to be out-of-date and thus should be
ignored. Proxies must be aware of outstanding observations and keep track of
the observe counters to make sure that out-of-order packets are dropped. I'm
not super happy about the mechanism, but that seems to be what the standard
has settled on. Passing everything through and letting the client sort it
out is explicitly not allowed.

[JLS] I cannot quite see that as being a true statement. It is true that
the most reason transmission must be considered the freshest, however a
client may want a more complete history of values than just the most recent
one and can deal with a non-freshest value. For example, if it is a
controller and is doing projections based on the current set of values,
having the "stale" value is still useful information even if it is not the
freshest. I do not see anything in RFC 7641 that says that out of order
packets are dropped by an intermediary. Are you saying this just based on
the fact that it is not freshest?

The dropping behavior is implied by two things:

* RFC7641 Section 5 says that proxies must track and renumber observe sessions
* draft-ietf-core-coap-tcp-tls-07 requires the observing endpoint to ignore the numeric value of the Observe header (because it is assumed to always be increasing).

I think the most reasonable way to reconcile these requirements is that an intermediary proxy MUST drop out-of-order observations. You could argue that such logic only applies when proxying to TLS/TCP/WebSockets, but I think the intent is for this behavior to be required independent of the raw transport.

I'm not saying that I like it or that I think that this is how it should have be, I'm just saying what the current documentation appears to require and what will likely be standardized.

-- RQ

Post by Jim Schaad
Jim
-- RQ

Post by Robert Quattlebaum
I have been sketching out what my TCP/UDP and UDP/TCP proxy code would
look like and I am having a problem deciding what should be done for
the observe option.
If you have the case of Client->UDP->Proxy->UDP->Server, then dealing
1. Pass them through unchanged or subtract a constant from the
observer value. This pushes the problem of out of order observes onto the

client.

Post by Robert Quattlebaum
The proxy would save the high water content.
2. Filter on observe value so that out of order transmissions are
suppressed.
3. Keep out of order items around for a while and see if the out of
order items show up and transmit them back as in order. I missing
items do not show up until the time out, then suppress the out of order

items.

Post by Robert Quattlebaum
4. Reorder the numbers so that things are seen in the wrong order by
the client. (Obviously a wrong answer.)
I assumed that 1 was the correct answer and deal with things that way.
Now I come to the question of Client->TCP->Proxy->UDP->Server. Since
the observe value is now to be ignored or removed, the first option
above will no longer work and I am not sure which the remaining
options is the correct one to do. Does a reliable transmission
protocol mean that we should suppress items?
Jim
_______________________________________________
core mailing list
https://www.ietf.org/mailman/listinfo/core

[Klaus Hartke]

Post by Jim Schaad
1. Pass them through unchanged or subtract a constant from the observer
value. This pushes the problem of out of order observes onto the client.
The proxy would save the high water content.

The sequence numbers in notifications are not end-to-end; they are
only meaningful between each pair of hops; each hop has to generate
its own sequence numbers [1]. It might be possible for a hop to
generate sequence numbers based on the values it receives, but my head
hurts when thinking about temporal sequence numbers, so I wouldn't
recommend it.

Post by Jim Schaad
2. Filter on observe value so that out of order transmissions are
suppressed.
3. Keep out of order items around for a while and see if the out of order
items show up and transmit them back as in order. I missing items do not
show up until the time out, then suppress the out of order items.

o It follows a best-effort approach for sending the current
representation to the client after a state change: clients should
see the new state after a state change as soon as possible, and
they should see as many states as possible.

So, depending on whether you optimize for "as soon as possible" or "as
many as possible", you should do 2 or 3. I would recommend 2.

This applies equally to reliable and unreliable transports.

Klaus

[1] https://tools.ietf.org/html/rfc7641#section-5
[2] https://tools.ietf.org/html/rfc7641#section-1.3