Why DCodec?

Message exchanges are a ubiquitous part of our daily lives and surroundings. All messages, from simple ones, such as signaling that you switched ON a light, to complex ones, such as a mobile device handover from one cell tower to another, are composed and parsed according to a definition.

Initially, a protocol structure is defined “casually” in your application data model (i.e. without a formal schema). Later, as the protocol grows in complexity, you move towards a more formal schema. This typically implies a more “rigid” way of handling messages, where deviations from the schema-defined structure are not acceptable. However, in many communication scenarios, you want to handle schema deviations without breaking the communication.

What if you could enjoy the best of both worlds, i.e., preserve the JSON-like flexibility you are used to, while taking advantage of the power of a schema? All this with an added bonus of an easy transition to binary, should you need it.

This is where OSS’ Dynamic Codec (DCodec) comes in. The DCodec offers:

  • A simple, yet powerful API that minimizes the amount of serialization code you need to write and maintain
  • Support for serializing/deserializing dynamic or user-provided C# objects
  • Data validation against a schema

To Code or Not to Code?

If you wonder whether you should develop your own encoder/decoder or use an existing one, consider the following.
"Any exploration program which "just happens" to include a new launch vehicle is, de facto, a launch vehicle program." Akin's Laws
Paraphrasing the above: "any project that includes writing an encoder/decoder, becomes, de facto, an encoder/decoder project".

  • Standardized schema support (ASN.1)
  • A smooth transition from schema-less to schema-based
  • Ability to decode only the message fields that are of interest
  • A powerful way to easily handle mismatches between the schema, the bindings, and the message (e.g., when messages are encoded with different versions of the schema), so that the application can handle both conflicts and unknown fields (log, mine for patterns, etc.)
  • An effortless way to switch between JSON and binary messages (ASN.1 Distinguished Encoding Rules)
  • 24x7 technical support.

IoT with the DCodec

OSS’ DCodec makes prototyping and deploying a communication software much easier. Here is a list of tools that may jumpstart and help us with our project. The tools are built around JSON, where the DCodec is one of them.

  1. Json2csharp - generates C# classes (aka bindings) from JSON.
  2. DCodec - can serialize JSON, allows great flexibility and data validation (against a schema), also can go binary (DER).
  3. Json2asn - creates a schema from a JSON message.

Defining our data

Let's say we develop an app that receives JSON messages to control a sensor. For simplicity we have just two kinds of messages: a sensor reading (outgoing) and sensor configuration (incoming).

Use json2csharp to create bindings.

Improve the bindings manually (just a bit).

{ "Message":"SensorConfiguration", "PublishPeriod":1.3 } { "Message":"SensorReading", "Value":123.4 }

public class RootObject { public string Message { get; set; } public double Value { get; set; } } public class RootObject { public string Message { get; set; } public double PublishPeriod { get; set; } }

public enum MessageType { SensorReading = 0, SensorConfiguration = 1 } public class SensorReading { public MessageType Message { get; set; } public double Value { get; set; } } public class SensorConfiguration { public MessageType Message { get; set; } public double PublishPeriod { get; set; } }


Now that we have defined the data that we want to exchange with our sensor, it's time to serialize them (send/receive). Encode and Decode is just one line of code and the message looks like this: { "Message":"SensorReading", "Value":1.22 }

var codec = new Oss.DCodec.JsonCodec() { }; var reading = new SensorReading() { Message = MessageType.SensorReading, Value = 1.22 }; var readingMessage = codec.Encode(reading); ... var decoded = codec.Decode<SensorReading>(readingMessage);

Adding a schema (for data validation)

What if we communicate with a sensor of an older model, which sends us data that do not conform to our model? It would be helpful to check its messages beyond just JSON well-formedness, i.e. to understand if the message has the expected fields and types. We can do this kind of validation with a schema, which defines the data structure, the field types and value ranges. The DCodec uses a schema, which can be obtained at JSON2ASN from the same JSON that we already learned how to communicate with. This is an ASN.1 schema. ASN.1 is an international standard used by many companies in telecom, avionics, intelligent transportation systems, etc, basically anywhere where robustness and security are important.

Most likely we’ll need to modify the schema manually, a little bit. In this case by replacing the UTF8String with a different type - ENUMERATED.

Next, we compile our schema into a key (or DKey) using this schema key generator. DKey can be loaded and used to validate our data by the DCodec. Every time when the message is encoded or decoded, it can be checked against the schema (say, for a missing a field or an integer range) so the DCodec will notify us of the invalid data.

GeneratedSchema DEFINITIONS AUTOMATIC TAGS ::= BEGIN SensorConfiguration ::= SEQUENCE { message UTF8String, publishPeriod REAL }; SensorReading ::= SEQUENCE { message UTF8String, value REAL } END

GeneratedSchema DEFINITIONS AUTOMATIC TAGS ::= BEGIN MessageType ::= ENUMERATED { SensorReading, SensorConfiguration } SensorConfiguration ::= SEQUENCE { message MessageType, publishPeriod REAL } SensorReading ::= SEQUENCE { message MessageType, value REAL }

string schemaKey = @"Zb5qQllrUFR..."; // omitted the entire DKey here var schema = new Oss.DCodec.DSchema(schemaKey); var type = schema.FindType("GeneratedSchema.SensorReading"); try { var decoded = codec.Decode<SensorReading>(readingMessage, type); } catch (Exception ex) { // invalid message }

Extending our model

Now let’s say we need to add new field in the sensor reading. Using the DCodec and schema there will be no problem with compatibility between our already deployed sensors and our newer sensors.

Devices that use the old schema will be able to decode the new messages (simply ignoring the parts that they do not understand) and vice versa.The DCodec can notify the app about any conflicts when there is a mismatch between the JSON, the schema and the bindings.This gives the DCodec great power in handling your communication with different versions of your devices and offers a flexible conflict resolution method by letting the app to choose whether to accept a message with deviations or not by responding to continue (DResponse.Continue) or to stop the decode process (DResponse.Break).

public enum MessageType { SensorReading = 0, SensorConfiguration = 1, SensorAlarm = 2 // added in v2 } public class SensorReading { public MessageType Message { get; set; } public double Value { get; set; } public int Timestamp { get; set; } // added in v2 } public class SensorConfiguration { public MessageType Message { get; set; } public double PublishPeriod { get; set; } } // added in v2 public class SensorAlarm { public enum AlarmCodes { hardwareFault, batteryLow, valueOutOfRange } public MessageType Message { get; set; } public AlarmCodes AlarmCode { get; set; } }

... try { var result = codec.DecodeDynamic(readingMessage, type, d => { if (d.Kind == DConflictKind.SchemaMissing) return DResponse.Continue; else return DResponse.Break; }); } catch { Console.WriteLine("Fatal conflict while decoding."); }

Data modeling, static vs dynamic

Depending on the scenario you are trying to address, it might be useful to pre-define a class-based model (a static type) and parse the JSON data directly into it, or it might be better to use dynamic types (created dynamically based on the input data). Static types are the preferred choice for cases when you need to use all the data, or rather when the ratio between the useful data and the total data is large. Static types offer the advantages of error checking at compile time and intellisense. Dynamic types, on the other, hand shine when used on large datasets, when you don’t need all the data and just want to extract some information from each structure in the JSON message, or where you don’t have the need or time to write up a class structure to fully model the data. Let’s look into that in a bit more detail.

Static Types

The most straightforward way to parse JSON data is to use a data model structure and bind the JSON message structure to it. This is very useful in situations where you know what data to expect and you don’t need to dynamically add new data, and also when you expect to consume all (or most) of the data. You can write our own data model by hand or you can use a series of freely-available online tools to generate a data model structure by giving an exhaustive set of JSON messages (the properties contained in the JSON messages must cover all the cases, otherwise the missing properties are not generated). In case you are not interested in all the data contained in the JSON message, you can omit or remove the members you don’t care about, and they will not be decoded from the JSON message.

Now we can imagine a control loop for multiple thermostats, using most of the data available to construct our logic, like this:

Take the example of the Nest Thermostat API (JSON message can be extracted from here). We know that for each thermostat we will have the information present. In this particular example, I used JSON2CSHARP to convert the message to a C# class structure. So we get the following model:

We can safely remove the generated members that we are not interested in, and the rest of the message will be properly decoded. For the purpose of this example, we are going to leave all the generated members from the message. Now, with the model defined, we can decode any Nest Thermostat JSON message by using a few lines of code:

The result will be a RootObject populated with all the data from the JSON message. We can now process the data, update it and re-encode it to JSON if needed simply by using the following line of code:

foreach (var thermostat in result.thermostats) { var t = thermostat.Value; if (t.can_heat) { if (t.ambient_temperature_c < t.target_temperature_c) { if (t.hvac_state != "heating") { // start heating } } } if (t.can_cool) { if (t.ambient_temperature_c > t.target_temperature_c) { if (t.hvac_state != "cooling") { // start cooling } if (t.has_fan && !t.fan_timer_active) { // start fan } } } }

public class InnerModel { public int humidity { get; set; } public string locale { get; set; } public string temperature_scale { get; set; } public bool is_using_emergency_heat { get; set; } public bool has_fan { get; set; } public string software_version { get; set; } public bool has_leaf { get; set; } public string where_id { get; set; } public string device_id { get; set; } public string name { get; set; } public bool can_heat { get; set; } public bool can_cool { get; set; } public string hvac_mode { get; set; } public double target_temperature_c { get; set; } public int target_temperature_f { get; set; } public int target_temperature_high_c { get; set; } public int target_temperature_high_f { get; set; } public int target_temperature_low_c { get; set; } public int target_temperature_low_f { get; set; } public double ambient_temperature_c { get; set; } public int ambient_temperature_f { get; set; } public int away_temperature_high_c { get; set; } public int away_temperature_high_f { get; set; } public double away_temperature_low_c { get; set; } public int away_temperature_low_f { get; set; } public string structure_id { get; set; } public bool fan_timer_active { get; set; } public string fan_timer_timeout { get; set; } public string name_long { get; set; } public bool is_online { get; set; } public string hvac_state { get; set; } } public class RootObject { public Dictionary<string, InnerModel> thermostats { get; set; } }

var codec = new Oss.Asn1DCodec.JsonCodec() { }; var result = codec.Decode<RootObject>(response);

var encoded = codec.Encode(result);

Dynamic Types

Dynamic types are very useful in the situation where you don’t care about all the data, don’t have a consistent schema for the data you are not interested in, or simply when you don’t need to write a class structure to model the parsed data. Take the following example: while parsing a Google Maps Places API to get all the business places around an app’s location (example can be found here), we get a lot of information in a JSON file. Most of it is not of interest to the application, so we need only to parse out and process the information we need. Using dynamic types (supported by the DCodec) we can very simply access only the properties we are interested in, and ignore the rest.

To print out all the restaurants that are open at this time around me, we could write the following lines of code:

By using this feature, we didn’t even have to write our own data model structure, and we can still access the data in a structured manner. What’s more, even if the data doesn’t have the exact same structure over more messages, we can still extract the data that is of interest to us, discarding the rest. What happens if one of the results does not contain the properties we are trying to access? By using the null conditional operators (opening_hours?.open_now), the code will not throw and work as expected: if opening_hours does not exist, the expression will evaluate to null. Adding data to the dynamic model is also very simple. Say we want to visit the restaurants whose names start with ‘A’ and we want to flag them. We can simply accomplish this as follows:

The serialized JSON will contain a new property for each restaurant that will be visited, called “toBeVisited”. We can add any type of data at any level of the JSON message. For instance, if we need to add an array of the names of all the restaurants that we will visit, this can be done as follows:

Also, we can now update any of the properties we know or add new properties, and re-encode the entire message with our changes by simply encoding the dynamic structure:

So, as you can see, the dynamic types are very versatile and can handle all the situations you might encounter while working with JSON messages with the least amount of effort.

var codec = new Oss.Asn1DCodec.JsonCodec() { }; dynamic decoded = codec.Decode(response); foreach (var result in decoded.results) { if (result.opening_hours?.open_now == true) Console.WriteLine($"{result.name} is open now"); }

foreach (var res in result.results) { if (res.opening_hours?.open_now == true && res.name?.StartsWith("A")) res.toBeVisited = true; }

s = new Oss.Asn1DCodec.DValueArray(); foreach (var res in result.results) { if (res.opening_hours?.open_now == true && res.name?.StartsWith("A")) res.toBeVisited = true; visitedRestaurants.Add(res.name); } result.visitedRestaurants = visitedRestaurants as dynamic;

var encoded = codec.Encode(decoded);