r/IOT • • 10d ago

Would a universal IoT message encoding/decoding library be useful?

Hi everyone,

I've been thinking about an idea for an open-source software library and I'd like to get some feedback from people working in IoT, embedded systems and industrial automation.

The problem I'm trying to address:

When developing IoT applications, integrating different devices often means dealing with different communication protocols, message structures, payload formats and vendor-specific implementations.

Every time a new device or protocol is introduced, developers may need to write custom parsers, encoders, decoders and translation logic.

This creates duplicated work, tightly coupled integrations and maintenance overhead.

My idea:

Build a reusable, protocol-independent library focused on encoding and decoding messages between software applications and IoT devices.

The general concept would be:

  • Receive a message from a device and decode it into a common, structured data model.

  • Take a command or structured message and encode it into the format expected by a specific device.

  • Support different protocols and payload formats through modular adapters or device-specific profiles.

  • Allow developers to add new device definitions without rewriting the application logic.

Essentially, the library would act as a bidirectional translation layer between IoT devices and applications.

For example, an application could send the same logical command regardless of whether the target device uses Modbus, MQTT, a proprietary binary protocol or another communication format.

The library would handle the translation.

I'd really appreciate your opinion on a few questions:

  1. Is this a real problem in your day-to-day IoT development work?

  2. Would a standalone library like this be useful, or do existing solutions already cover this need adequately?

  3. Would you prefer a common data model with device-specific codecs, or a more protocol-oriented architecture?

  4. Which protocols and formats would be essential for an initial implementation?

  5. Would you consider using or contributing to an open-source project like this?

I'm not trying to build another IoT platform, broker or device management solution.

The goal is much narrower: simplify the encoding and decoding of messages between heterogeneous IoT devices and software systems.

I'm interested in honest technical feedback, including reasons why this might not be useful.

Thanks!

10 Upvotes

17 comments sorted by

7

u/Panometric 10d ago

Being universal is hard, usually at a big price. Check out protocol buffers and flat buffers. They create custom protocols from Meta data and generate coders in many languages.

1

u/JacCroce 10d ago

Do you think Protocol Buffers or FlatBuffers could also be used to map devices connected to platforms like The Things Network (TTN)?

2

u/Panometric 10d ago

u/JacCroce I don't understand the question. They are just a way to encapsulate and encode structured data like time stamped sensor readings. I have used them mostly with MQTT. Flatbuffers works for C and many other languages , is very efficient, and microcontroller sized.

2

u/mfalkvidd 10d ago

I use Kaitai for this. Works well, imho.

0

u/JacCroce 10d ago

I'm not familiar with Kaitai. Could you send me some documentation or a link to get started?

8

u/setuid_w00t 10d ago

1

u/Best-Leave6725 9d ago

I haven't looked at XKCD in years but this came straight to my mind when I saw this thread.

4

u/Minimum-Sprinkles843 10d ago

No, period. Many years ago I was working on a smart city project that involved communication with lots of LoRaWAN devices and I had thought about the same thing. During development, it quickly became obvious that such a thing is impossible even within one org.

1

u/Jetta4 9d ago

Yes, even the same LoRaWAN device model can need different decoders based on firmware version.

Universal doesn’t work, some things need to be set at individual device level.

2

u/rsclient 9d ago

No, and here's why: we already have a bunch of simple, straightforward standards. And the number of IOT devices that use them is pitifully small.

For example, out of 4 bluetooth blood-pressure cuffs I checked, each one used their own custom protocol, and none used the simple BT bloodpressure protocol that's already available.

2

u/xaviertech 5d ago

From the LoraWan side the hard part isn't really the decoding logic, it's that every vendor does its own payload format and it changes between firmware versions... sometimes even the same meter model has different frames. The TTN device repository already solves part of it with JS decoders per device. If you build something I'd focus on a shared format for the decoded output (units, names) more than on the decoding itself, that's where I always end up writing glue code

2

u/TheProffalken 10d ago

I'd also check out The Things Network device catalogue as it already has a lot of JavaScript to convert from the various manufacturers to some kind of common format.

The other thing to look at would be Weaver from the Cloud Native Computing Foundation - it's designed for open telemetry based observability, but there's nothing to stop you using it as a way to create and document custom schemas for log and metric names and attributes depending on the platforms you're using?

1

u/patrolsnlandrcuisers 10d ago

Honestly don't bother man been down this rabbit hole myself seen a bunch of attempts at this over the years, device makers are always changing shit so your tool will just lag out of date or hit edge cases and piss of ure users, the way ai trends these days its one prompt to translate most things and then ure getting the info directly from the source the people who made it, hopefully matter will help on this front in general 

1

u/konacurrents 10d ago

At the message layer, JSON is rather universal. Then it’s up to apps to determine what to do with it. And JSON (string) can be sent through various communication protocols (eg MQTT, BLE, REST)

1

u/clarksonswimmer 9d ago

Isn’t that what Matter is supposed to be solving?

1

u/PeterHumaj 1d ago

Google protobufs? Usedg dy gRPC, or MQTT Sparkplug...