[core] New CoAP Options on the horizon - Accept-Content-Format-Version and Content-Format-Version

[Alexander Pelov]

Dear all,

IANA has received a request from Michael Koster to register two new CoAP options:
- Accept-Content-Format-Version
- Content-Format-Version

The registration happens after an expert review and after a couple of mail exchanges and meeting Michael in person in Chicago, we (the experts and Michael) decided that the discussion should be brought to the mailing list, before getting the final approval.

One of the major questions that needs to be discussed is the following:
What are the reasons *NOT* to allocate a new Content-Format number (RFC 7252) for each new version? (in the HTTP world, "version" is a simple parameter to the Content-Type, so why not treat it as such?)

The question is not trivial, as it touches upon content format negotiation, e.g. how should the Server use the Accept-Content-Format-Version in order to indicate that it is willing to accept both application/vnd.ocf+cbor AND application/application/vnd.ocf+cbor;version=2.0, with a preference for the latter? In the semantics of the version parameter, the two should be considered completely independent/incompatible types.

The current way of doing this would be to allocate separate Content-Format numbers to "application/application/vnd.ocf+cbor" (10000 as it stands now) and "application/application/vnd.ocf+cbor;version=2.0" (e.g. 10001).. and then use some form of content negotiation.

This however can have its disadvantages, .. and the way industry does use versioning in some cases (e.g. people DO use the version parameter to version their RESTful APIs.. which means, that NOT having such mechanism in CoAP could theoretically drive them to re-register blocks of Content-Format numbers, just to differentiate between their APIs).

In any case, we need to have this discussion documented on the mailing list (as previously discussed with Michael), so here we go..

Best,
Alexander

[Klaus Hartke]

Post by Alexander Pelov
- Accept-Content-Format-Version
- Content-Format-Version

Spec?
What are the other questions?

Post by Alexander Pelov
What are the reasons *NOT* to allocate a new Content-Format number (RFC
7252) for each new version? (in the HTTP world, "version" is a simple
parameter to the Content-Type, so why not treat it as such?)

None?

Post by Alexander Pelov
The question is not trivial, as it touches upon content format negotiation,
e.g. how should the Server use the Accept-Content-Format-Version in order
to indicate that it is willing to accept both application/vnd.ocf+cbor AND
application/application/vnd.ocf+cbor;version=2.0, with a preference for the
latter? In the semantics of the version parameter, the two should be
considered completely independent/incompatible types.

CoAP currently has no features for content format negotiation, only
for the selection of an available content format. If a server wishes
to indicate available content formats, it needs to do so in the
(hypermedia) representation. (The CoAP Accept option can only be used
in requests, not in responses.)

Are the two options a proposal to change that?

Need spec.

Post by Alexander Pelov
The current way of doing this would be to allocate separate Content-Format
numbers to "application/application/vnd.ocf+cbor" (10000 as it stands now)
and "application/application/vnd.ocf+cbor;version=2.0" (e.g. 10001).. and
then use some form of content negotiation.

Yes.

Post by Alexander Pelov
This however can have its disadvantages, .. and the way industry does use
versioning in some cases (e.g. people DO use the version parameter to
version their RESTful APIs.. which means, that NOT having such mechanism in
CoAP could theoretically drive them to re-register blocks of Content-Format
numbers, just to differentiate between their APIs).

Working as intended.

Post by Alexander Pelov
In any case, we need to have this discussion documented on the mailing list
(as previously discussed with Michael), so here we go..

I'm sure Michael has analyzed the problem, evaluated different options
to solve the problem, and contacted the experts with the best solution
he could find. However, since the identifier spaces in CoAP are so
small and the unrestricted addition of new CoAP features can lead to
fragmentation, every addition must be carefully considered. For that,
it is really helpful to establish an understanding of the problem
before pushing a particular solution.

Klaus

[Alexander Pelov]

Hi Klaus,

Post by Klaus Hartke

Post by Alexander Pelov
- Accept-Content-Format-Version
- Content-Format-Version

Spec?
What are the other questions?

The other questions are mostly details that we can (easily) figure out once the major one is answered. For example, should the Accept-Content-Format-Version be Repeatable or Not? (I don’t want to get to more details on this for the moment - and they will surely come up if necessary).

Post by Klaus Hartke

Post by Alexander Pelov
What are the reasons *NOT* to allocate a new Content-Format number (RFC
7252) for each new version? (in the HTTP world, "version" is a simple
parameter to the Content-Type, so why not treat it as such?)

None?

Post by Alexander Pelov
The question is not trivial, as it touches upon content format negotiation,
e.g. how should the Server use the Accept-Content-Format-Version in order
to indicate that it is willing to accept both application/vnd.ocf+cbor AND
application/application/vnd.ocf+cbor;version=2.0, with a preference for the
latter? In the semantics of the version parameter, the two should be
considered completely independent/incompatible types.

CoAP currently has no features for content format negotiation, only
for the selection of an available content format. If a server wishes
to indicate available content formats, it needs to do so in the
(hypermedia) representation. (The CoAP Accept option can only be used
in requests, not in responses.)
Are the two options a proposal to change that?
Need spec.

Yes, you are right indeed. The *Client* indicates in its CoAP Accept option the Content-Format it is willing to accept. Now, if you have the same Content-Format, but with different versions, then you could imagine having something like:
Accept: 10000 (which maps to application/application/vnd.ocf+cbor)
Accept-Content-Format-Version: 2.0

Instead of:
Accept: 10001 (application/application/vnd.ocf+cbor;version=2.0 which would need to be registered to 10001 for example)

The intent here is to always indicate "10000" in your hypermedia links, and without changing them to be able to upgrade/downgrade the version of the content-format (e.g. I like to think of this as changing APIs - you upgrade a Client, and then it is on ver2 - but still using the original Content-Format, but this may not be the only reason Michael proposes these options).

Post by Klaus Hartke

Post by Alexander Pelov
The current way of doing this would be to allocate separate Content-Format
numbers to "application/application/vnd.ocf+cbor" (10000 as it stands now)
and "application/application/vnd.ocf+cbor;version=2.0" (e.g. 10001).. and
then use some form of content negotiation.

Yes.

Post by Alexander Pelov
This however can have its disadvantages, .. and the way industry does use
versioning in some cases (e.g. people DO use the version parameter to
version their RESTful APIs.. which means, that NOT having such mechanism in
CoAP could theoretically drive them to re-register blocks of Content-Format
numbers, just to differentiate between their APIs).

Working as intended.

Post by Alexander Pelov
In any case, we need to have this discussion documented on the mailing list
(as previously discussed with Michael), so here we go..

I'm sure Michael has analyzed the problem, evaluated different options
to solve the problem, and contacted the experts with the best solution
he could find. However, since the identifier spaces in CoAP are so
small and the unrestricted addition of new CoAP features can lead to
fragmentation, every addition must be carefully considered. For that,
it is really helpful to establish an understanding of the problem
before pushing a particular solution.

I agree with you on this. Having this understanding - and documented on the mailing list - is the intent of this discussion.

Alexander

Post by Klaus Hartke
Klaus
_______________________________________________
core mailing list
https://www.ietf.org/mailman/listinfo/core

[Kovatsch Matthias]

Hi all

I want to point out that this is related to the parameter option discussion we had for a long time: https://tools.ietf.org/html/draft-doi-core-parameter-option-03
The work was dropped because there was too much resistance to include this feature (it least in my understanding).

Coming from the design guidelines discussed in T2TRG, it is expected that versioning is handled within the representation format (while a good design also ensures backward and forward compatibility). If a representation format is not compatible with previous versions, it is a different format that needs a new media type, and hence CoAP identifier.
(Note that these guidelines have not reached consensus, and hence cannot be referenced. Please add other views; it might be a valuable T2TRG discussion.)

Ciao
Matthias

-----Original Message-----
Sent: Sunday, 9 April 2017 22:01
Subject: Re: [core] New CoAP Options on the horizon - Accept-Content-Format-
Version and Content-Format-Version
Hi Klaus,

Post by Klaus Hartke

Post by Alexander Pelov
IANA has received a request from Michael Koster to register two new
CoAP
- Accept-Content-Format-Version
- Content-Format-Version

Spec?
What are the other questions?

The other questions are mostly details that we can (easily) figure out once the
major one is answered. For example, should the Accept-Content-Format-Version
be Repeatable or Not? (I don’t want to get to more details on this for the
moment - and they will surely come up if necessary).

Post by Klaus Hartke

Post by Alexander Pelov
What are the reasons *NOT* to allocate a new Content-Format number (RFC
7252) for each new version? (in the HTTP world, "version" is a simple
parameter to the Content-Type, so why not treat it as such?)

None?

Post by Alexander Pelov
The question is not trivial, as it touches upon content format
negotiation, e.g. how should the Server use the
Accept-Content-Format-Version in order to indicate that it is willing
to accept both application/vnd.ocf+cbor AND
application/application/vnd.ocf+cbor;version=2.0, with a preference
for the latter? In the semantics of the version parameter, the two should be

considered completely independent/incompatible types.

Post by Klaus Hartke
CoAP currently has no features for content format negotiation, only
for the selection of an available content format. If a server wishes
to indicate available content formats, it needs to do so in the
(hypermedia) representation. (The CoAP Accept option can only be used
in requests, not in responses.)
Are the two options a proposal to change that?
Need spec.

Yes, you are right indeed. The *Client* indicates in its CoAP Accept option the
Content-Format it is willing to accept. Now, if you have the same Content-
Format, but with different versions, then you could imagine having something
Accept: 10000 (which maps to application/application/vnd.ocf+cbor)
Accept-Content-Format-Version: 2.0
Accept: 10001 (application/application/vnd.ocf+cbor;version=2.0 which would
need to be registered to 10001 for example)
The intent here is to always indicate "10000" in your hypermedia links, and
without changing them to be able to upgrade/downgrade the version of the
content-format (e.g. I like to think of this as changing APIs - you upgrade a Client,
and then it is on ver2 - but still using the original Content-Format, but this may
not be the only reason Michael proposes these options).

Post by Klaus Hartke

Post by Alexander Pelov
The current way of doing this would be to allocate separate
Content-Format numbers to "application/application/vnd.ocf+cbor"
(10000 as it stands now) and
"application/application/vnd.ocf+cbor;version=2.0" (e.g. 10001).. and then

use some form of content negotiation.

Post by Klaus Hartke
Yes.

Post by Alexander Pelov
This however can have its disadvantages, .. and the way industry does
use versioning in some cases (e.g. people DO use the version
parameter to version their RESTful APIs.. which means, that NOT
having such mechanism in CoAP could theoretically drive them to
re-register blocks of Content-Format numbers, just to differentiate between

their APIs).

Post by Klaus Hartke
Working as intended.

Post by Alexander Pelov
In any case, we need to have this discussion documented on the
mailing list (as previously discussed with Michael), so here we go..

I'm sure Michael has analyzed the problem, evaluated different options
to solve the problem, and contacted the experts with the best solution
he could find. However, since the identifier spaces in CoAP are so
small and the unrestricted addition of new CoAP features can lead to
fragmentation, every addition must be carefully considered. For that,
it is really helpful to establish an understanding of the problem
before pushing a particular solution.

I agree with you on this. Having this understanding - and documented on the
mailing list - is the intent of this discussion.
Alexander

Post by Klaus Hartke
Klaus
_______________________________________________
core mailing list
https://www.ietf.org/mailman/listinfo/core

_______________________________________________
core mailing list
https://www.ietf.org/mailman/listinfo/core

[Michael Koster]

Hi,

This gets to the core of the problem as I see it.

It is important to be able to construct a request header from the information in a hypermedia control. For links to CoAP resources, we have the content-format number. Even if we have content-format negotiation in CoAP, there is still application awareness of content formats.

The purpose of the content format *version* is to indicate changes in the serialization that don't impact the application compatibility, which the system can handle transparently if it has the correct serializer.

One use case is to align the representations in OCF with standard IETF content-formats over time, as we are doing with CoRE link-format (RFC6690). Doing this may require a linear update to the OCF content format, which already has a compatible data model, in addition to the implicit support of a standard content format. If there is a move to SenML serialization, for example (hypothetical), but uisng the same OCF data model, then we will need to have an updated OCF content-format version to track OCF versions, even though by then the server may also be able to returnin link-format+cbor (TBD) and senml+cbor (TBD) content format numbers, because of existing OCF applications that are aware of the content-format number.

Thus it is desireable to keep the link attribute value consistent to avoid having to change application layer logic, which is aware of content format numbers and may query a collection using content-format numbers, etc.

Does this make sense?

Best regards,

Michael

Post by Alexander Pelov
Accept: 10000 (which maps to application/application/vnd.ocf+cbor)
Accept-Content-Format-Version: 2.0
Accept: 10001 (application/application/vnd.ocf+cbor;version=2.0 which would need to be registered to 10001 for example)
The intent here is to always indicate "10000" in your hypermedia links, and without changing them to be able to upgrade/downgrade the version of the content-format (e.g. I like to think of this as changing APIs - you upgrade a Client, and then it is on ver2 - but still using the original Content-Format, but this may not be the only reason Michael proposes these options).

Post by Klaus Hartke

Post by Alexander Pelov
The current way of doing this would be to allocate separate Content-Format
numbers to "application/application/vnd.ocf+cbor" (10000 as it stands now)
and "application/application/vnd.ocf+cbor;version=2.0" (e.g. 10001).. and
then use some form of content negotiation.

Yes.

Post by Alexander Pelov
This however can have its disadvantages, .. and the way industry does use
versioning in some cases (e.g. people DO use the version parameter to
version their RESTful APIs.. which means, that NOT having such mechanism in
CoAP could theoretically drive them to re-register blocks of Content-Format
numbers, just to differentiate between their APIs).

Working as intended.

Post by Alexander Pelov
In any case, we need to have this discussion documented on the mailing list
(as previously discussed with Michael), so here we go..

I'm sure Michael has analyzed the problem, evaluated different options
to solve the problem, and contacted the experts with the best solution
he could find. However, since the identifier spaces in CoAP are so
small and the unrestricted addition of new CoAP features can lead to
fragmentation, every addition must be carefully considered. For that,
it is really helpful to establish an understanding of the problem
before pushing a particular solution.

[Carsten Bormann]

I think that this is indicative of a more general problem. Media types have various conventions for indicating subclass relationships: the two-part notation, structured syntax suffixes, but no general way to indicate that a representation is both within a larger and a more specific type. Content-format makes this slightly worse by hiding both the special-case mechanisms cited above and media type parameters, which are another subclassing mechanism. A fix might be too be able to indicate multiple content-formats. But this requires a bit more thinking.

Sent from mobile

Post by Michael Koster
Does this make sense?

[Robert Quattlebaum]

It is a bit of a long shot, but may be worth bringing up...

Apple came up with a hierarchical content-type mechanism they called the "Universal Type Identifier":

https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/understanding_utis/understand_utis_conc/understand_utis_conc.html <https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/understanding_utis/understand_utis_conc/understand_utis_conc.html>

This itself doesn't solve the problem, but if there was a hierarchical registry for assigning numeric identifiers at each node (OID style), then the file type could be specified as a sequence of numbers. Applications would then only need to verify that the file type has the appropriate prefix (if well designed, this could just be an opaque string of bytes) and ignore anything they don't understand after.

For example, let's assume the registry has some classes like this:

public.data
0: public.text
0: public.html
1: public.plain-text
0: net.daringfireball.markdown
2: public.json
0: com.example.iot-foo
1: public.image
0: public.jpeg
1: public.tiff

So, for the content type com.example.iot-foo, the type would be encoded as "0.2.0". A TIFF image would be "1.1".

If a browser has a nice JSON pretty-printer but has no idea what com.example.iot-foo is, then it would just check for the "0.2" type prefix, ignoring anything after.

The problem is that the UTI concept never seemed to quite make it to the IETF, or any other standards body (even though it was seemingly designed in such a way as to make standardization somewhat easy). Nonetheless, the concept above might be useful to consider if we are actually thinking about digging up this can of worms.

If we go this route, perhaps some sort of CBOR encoding would be appropriate.

Realistically, this is a big change, if taken as described. I wonder if there is a way to build this concept on top of what we already have?

-- RQ

Post by Carsten Bormann
I think that this is indicative of a more general problem. Media types have various conventions for indicating subclass relationships: the two-part notation, structured syntax suffixes, but no general way to indicate that a representation is both within a larger and a more specific type. Content-format makes this slightly worse by hiding both the special-case mechanisms cited above and media type parameters, which are another subclassing mechanism. A fix might be too be able to indicate multiple content-formats. But this requires a bit more thinking.
Sent from mobile

Post by Michael Koster
Does this make sense?

_______________________________________________
core mailing list
https://www.ietf.org/mailman/listinfo/core

[Klaus Hartke]

So something like this?


+-- <root>
|
+-- A. content coding 'identity'
| |
| +-- A.0. text/plain; charset=utf-8
| |
| +-- A.40. application/link-format
| |
| +-- A.41. application/xml
| | |
| | +-- A.41.65001. application/senml+xml
| | +-- A.41.65002. application/sensml+xml
| |
| +-- A.42. application/octet-stream
| |
| +-- A.47. application/exi
| | |
| | +-- A.47.65003. application/senml+exi
| | +-- A.47.65004. application/sensml+exi
| |
| +-- A.50. application/json
| | |
| | +-- A.50.51. application/json-patch+json
| | +-- A.50.52. application/merge-patch+json
| | +-- A.50.256. application/coap-group+json
| | +-- A.50.11543. application/vnd.oma.lwm2m+json
| | +-- A.50.65005. application/senml+json
| | +-- A.50.65006. application/sensml+json
| |
| +-- A.60. application/cbor
| | |
| | +-- A.60.16. application/cose; cose-type="cose-encrypt0"
| | +-- A.60.17. application/cose; cose-type="cose-mac0"
| | +-- A.60.18. application/cose; cose-type="cose-sign1"
| | +-- A.60.96. application/cose; cose-type="cose-encrypt"
| | +-- A.60.97. application/cose; cose-type="cose-mac"
| | +-- A.60.98. application/cose; cose-type="cose-sign"
| | +-- A.60.101. application/cose-key
| | +-- A.60.102. application/cose-key-set
| | +-- A.60.10000. application/vnd.ocf+cbor
| | +-- A.60.65007. application/senml+cbor
| | +-- A.60.65008. application/sensml+cbor
| |
| +-- A.11542. application/vnd.oma.lwm2m+tlv
|
+-- B. content coding 'deflate'
| |
| (same structure as 'identity')
|
+-- C. content coding 'gzip'
| |
| (same structure as 'identity')
|
+-- D. content coding 'aes128gcm'
| |
| (same structure as 'identity')
|
+-- E. content coding 'exi'
|
+-- E.65003. application/senml+exi
+-- E.65004. application/sensml+exi


Michael: Does your use case fit in there?

Klaus

[Carsten Bormann]

Now, is this a hierarchy?

Post by Klaus Hartke
| | +-- A.41.65001. application/senml+xml
| | +-- A.47.65003. application/senml+exi
| | +-- A.50.65005. application/senml+json
| | +-- A.60.65007. application/senml+cbor
+-- E.65003. application/senml+exi

The “senml”-ness could be another node in the tree, with leaves for the cases listed here.

Grüße, Carsten

[Klaus Hartke]

Post by Carsten Bormann
Now, is this a hierarchy?

If you want it to be a hierarchy, then there are two options:


+-- <root>
|
+-- data model X
| |
| +-- cbor
| +-- exi
| +-- json
| +-- xml
|
+-- data model Y
|
+-- cbor
+-- exi
+-- json
+-- xml


+-- <root>
|
+-- cbor
| |
| +-- data model X
| +-- data model Y
|
+-- exi
| |
| +-- data model X
| +-- data model Y
|
+-- json
| |
| +-- data model X
| +-- data model Y
|
+-- xml
|
+-- data model X
+-- data model Y


The second allows a processor to fall back to generic
cbor/exi/json/xml if it does not understand data model X/Y. Does the
first have a similar property?

Klaus

[Carsten Bormann]

Post by Klaus Hartke
The second allows a processor to fall back to generic
cbor/exi/json/xml if it does not understand data model X/Y. Does the
first have a similar property?

If you are about senml, but don’t care about a specific serialization, the other hierarchy would make more sense.

(I’m not sure that this is usefully a hierarchy — I just saw the depictions as a tree and the question was my knee jerk reaction to seeing a tree.)

Grüße, Carsten

[Klaus Hartke]

+---------------+---------------+---------------+
/| /| /| /|
/ | / | / | / |
data / | / | / | / |
model Y +---------------+---------------+---------------+ |
| | | | | | | |
| | | | | | | |
| +-----------|---+-----------|---+-----------|---+ deflate
| / | / | / | /
| / | / | / | /
data |/ |/ |/ |/
model X +---------------+---------------+---------------+ identity
cbor exi json xml


Better?

Klaus

Post by Carsten Bormann

Post by Klaus Hartke
The second allows a processor to fall back to generic
cbor/exi/json/xml if it does not understand data model X/Y. Does the
first have a similar property?

If you are about senml, but don’t care about a specific serialization, the other hierarchy would make more sense.
(I’m not sure that this is usefully a hierarchy — I just saw the depictions as a tree and the question was my knee jerk reaction to seeing a tree.)
Grüße, Carsten