Bridging OPC UA to MQTT and Kafka, A 2024 Playbook

TL;DR — Subscribe with sensible deadbands, normalize the address space at the bridge, publish to MQTT for live consumers and stream to Kafka for durable replay. Don’t do it in the broker.

OPC UA is the dominant industrial protocol on the OT side in 2024. It’s a real spec, with security, browsing, subscriptions, and a sane object model. The bad news is that the address spaces vendors expose are wildly different in shape, and you almost never want to hand them to your IT data platform raw. The bridge from OPC UA into MQTT and Kafka is where you do the translation that keeps both sides happy.

This post is the playbook I use. C with open62541 1.4 for the bridge itself, because it’s stable, well-maintained, and the licensing works for commercial deployments. MQTT 5.0 for the live north-bound feed. Kafka 3.7 for the durable log. We’ll cover subscription design, schema normalization, security, and what to put where.

A philosophical note. The bridge owns the canonical tag namespace. If you let each consumer carry its own naming convention, you’ll spend years untangling it. Decide the namespace once, at the bridge, and enforce it.

Subscription Design

OPC UA subscriptions are the right way to read from a server. Polling individual nodes does work and is sometimes necessary for older servers, but a proper subscription with publishing intervals and deadbands is dramatically more efficient on the wire and on the server CPU.

For each monitored item, set:

• A publishing interval that matches how often you actually need the value. 250 ms for fast loops, 1 s or 5 s for slower process tags.
• A queue size large enough to survive a brief network hiccup, usually 10 to 20.
• A dataChangeFilter with a deadband. Absolute deadband for engineering units, percent deadband for ratios. This is the single biggest win for wire efficiency.

UA_MonitoredItemCreateRequest req = UA_MonitoredItemCreateRequest_default(nodeId);
req.requestedParameters.samplingInterval = 100.0;     // ms
req.requestedParameters.queueSize = 10;
req.requestedParameters.discardOldest = true;

UA_DataChangeFilter filter;
UA_DataChangeFilter_init(&filter);
filter.trigger = UA_DATACHANGETRIGGER_STATUSVALUE;
filter.deadbandType = UA_DEADBANDTYPE_ABSOLUTE;
filter.deadbandValue = 0.1;                           // engineering units

UA_ExtensionObject_setValue(&req.requestedParameters.filter,
    &filter, &UA_TYPES[UA_TYPES_DATACHANGEFILTER]);

The deadband alone often cuts message volume by 70–90 percent on analog signals that wiggle around a setpoint.

Normalizing the Address Space

Most OPC UA servers expose tags via vendor-specific node IDs. Something like ns=4;s=|var|MyController.GVL.Line1.Pressure1.Bar. That’s fine for the OT side. It’s terrible to publish into your data platform.

The bridge maintains a mapping table, configured per server, that translates the OPC UA node identity into a canonical topic and a metric ID:

mappings:
  - node: "ns=4;s=|var|MyController.GVL.Line1.Pressure1.Bar"
    topic: "plant/jkt1/packaging/line1/press1/bar"
    metric_id: 1024
    units: "bar"
    deadband: 0.05
  - node: "ns=4;s=|var|MyController.GVL.Line1.Temp1.C"
    topic: "plant/jkt1/packaging/line1/temp1/c"
    metric_id: 1025
    units: "degC"
    deadband: 0.2

The mapping file is the contract. When a tag is added or renamed in the PLC, the mapping is updated. The downstream world only sees the canonical topic. Versions of the mapping file are themselves committed to git, reviewed, and deployed. This is the most important operational discipline in the whole bridge.

Publishing to MQTT

Live consumers, SCADA dashboards, HMIs, alarm engines, want the latest value with low latency. MQTT is the right transport. The bridge publishes each accepted change to the canonical topic with QoS 1 and a small payload.

Payloads should be self-describing enough to be useful but small enough to be cheap. I use compact JSON for sites with bandwidth to spare, MessagePack or protobuf for cellular. Always include the OPC UA source timestamp, server timestamp, status code, and the value:

{"t":1723793400123,"st":1723793400125,"q":192,"v":72.4,"u":"degC"}

The status code matters more than people think. OPC UA’s quality flags distinguish “good”, “uncertain” and “bad”, with sub-codes for why. Carry it through to MQTT and let consumers decide what to do with uncertain readings.

Streaming to Kafka

Kafka is for durable replay, fan-out, and any consumer that wants more than the latest value. The bridge sends a parallel record per accepted change to a Kafka topic, keyed by device or line for partition-stable ordering.

For schema, Avro through Schema Registry or protobuf both work. The payload mirrors the MQTT one but includes the canonical metric ID and the source mapping version, so downstream consumers can detect mapping changes:

{
  "ts_ns": 1723793400123000000,
  "metric_id": 1024,
  "value": 72.4,
  "status": 192,
  "mapping_version": "2024-08-16-01"
}

mapping_version is what saves you when somebody renames a tag at the PLC and ten minutes of downstream analytics turn into nonsense. Detect the mapping version change in your stream processor and either backfill or alert.

Security

OPC UA security is not optional in 2024. Use the Basic256Sha256 or Aes256_Sha256_RsaPss security policies, both in SignAndEncrypt mode, with certificates issued from your internal CA. The bridge’s certificate is trusted by the server, and vice versa. Plain None security on a control network is asking for trouble.

UA_ClientConfig *cfg = UA_Client_getConfig(client);
UA_ClientConfig_setDefaultEncryption(cfg,
    certificate, privateKey, NULL, 0, NULL, 0);
cfg->securityMode = UA_MESSAGESECURITYMODE_SIGNANDENCRYPT;
cfg->securityPolicyUri = UA_STRING_ALLOC(
    "http://opcfoundation.org/UA/SecurityPolicy#Basic256Sha256");

On the MQTT side, TLS to the broker, with client certificates if your fleet management supports it. On the Kafka side, SASL/SCRAM at minimum and mTLS if you’re crossing the OT/IT boundary inside the bridge.

Sizing

A single bridge process on a small VM, 2 vCPU and 2 GB RAM, comfortably handles 50k monitored items at typical industrial rates. open62541 is efficient and the per-item overhead is small. Beyond that, run multiple bridge processes, one per OPC UA server or one per logical area, and let them share a Kafka topic.

The bottleneck you’ll hit first is not CPU. It’s the OPC UA server’s own subscription limits. Many PLCs cap concurrent monitored items per session at a few thousand. Split sessions accordingly.

Common Pitfalls

• Polling instead of subscribing. If the server supports subscriptions, use them. Polling is wasteful and won’t scale.
• No deadband. Analog signals jitter. Without deadbands, you’ll publish noise.
• Trusting source timestamps blindly. PLC clocks drift. Capture both source and server timestamps and reconcile downstream.
• Publishing into the MQTT broker’s own retained store at high rates. Retained is for last-known state, not history. Use a separate “last known value” service if you need fast LKV lookup.
• No backpressure handling. If Kafka is slow, the bridge’s send buffer grows. Without backpressure, you’ll either lose data or OOM. Block on the OPC UA subscription side when Kafka can’t keep up.

Wrapping Up

The OPC UA bridge is the seam where OT meets IT. Build it carefully and the rest of the platform is calm. Subscribe efficiently, normalize at the bridge, publish to MQTT for live consumers, stream to Kafka for everything else, secure everything, version your mappings.

For the broader context, the IIoT reference architecture post shows where this sits. The open62541 documentation is the canonical reference for the client API used here.

Bridges built well disappear into the wallpaper. That’s the goal.