Emutex developed the ubiworx™ IoT framework to allow rapid development and deployment of IIoT solutions by providing a rich set of protocol handlers, enhanced end to end security and comprehensive device management capabilities to edge devices in an IoT network. This complements many other IoT frameworks which tend to be cloud focused and omit important aspects such as protocol translation, data normalization, sensor configuration and sensor management. ubiworx™ enables IIoT customers to rapidly develop a complete end-to-end solution that takes data from sensor to cloud without, in many cases, having to develop any software.

 

Executive Summary:

Our client approached us to build a low cost IIoT solution to run a trial monitoring a number of sensors in their factory. They needed an off the shelf solution that enabled cost effective data acquisition from existing and new sensors over a network that could gather data, perform local analytics, do alarm and event generation and export to the corporate SCADA system. Our recommendation was to deploy ubiworx™ as a turn-key, cost-effective software solution. We installed ubiworx™ on a network of Supermicro E100-8Q gateways running WindRiver Linux. The system was deployed and data was gathered for analysis.

In this paper, we describe the customer experience of deploying ubiworx™ in a production environment, the challenges that were faced, the benefits and advantages of using the ubiworx™ framework for IoT gateways and the opportunities for further development that were presented. Several interesting discoveries were made which are also outlined in this paper.

 

Introduction

The Internet of Things (IoT) is a network of physical objects that utilise embedded technology to communicate and interact with the external environment via the Internet. Due to the huge number of connected devices and the associated revenue potential, the IoT is of enormous interest for all the large industry players in the Internet area including Cisco, Intel, GE, ARM, Analog Devices, etc.

The Industrial IoT (IIoT) is the application of IoT technology to manufacturing industries. IIoT harnesses existing sensor data and incorporates big data technology and M2M communications to build on automation technologies that have been in place for many years. Smart machines are better than humans at capturing and communicating sensor data and the harvested data can be visualized and mined for correlations leading to improved efficiencies.

Traditionally, industrial monitoring and SCADA systems have been expensive, proprietary, closed and inflexible. Using modern low cost hardware for gateways and sensors and Platform as a Service (PaaS) software licensing allows huge cost savings and flexibility for data acquisition.

Emutex developed the ubiworx™ IoT framework to allow rapid development and deployment of IIoT solutions by providing a rich set of protocol handlers, enhanced end to end security and comprehensive device management capabilities to edge devices in an IoT network. This complements many other IoT frameworks which tend to be cloud focused and omit important aspects such as protocol translation, data normalization, sensor configuration and sensor management. ubiworx™ enables IIoT customers to rapidly develop a complete end-to-end solution that takes data from sensor to cloud without, in many cases, having to develop any software.

 

Client Requirements

Cost Effective Gateway Hardware

The gateway hardware needed to be cost effective, but still be industrial specification. The Supermicro E100-8Q IoT Gateway System was chosen as the gateway hardware. The E100-8Q has built in digital GPIO pins, 8 channels of analog input, RS485 interface and PWM output allowing it to be directly connected to a wide variety of passive sensors to monitor many industrial parameters. The E100-8Q utilizes an Intel® Quark CPU and is an Intel® approved Moon Island gateway running the Windriver Linux operating system. It has 512MB RAM and 16GB flash disk to accommodate demanding software applications.

 

Cost Effective Sensors

The requirement for the sensors was that they should be passive if possible and not require any additional I/O module or other adapters. 4-20mA sensors were used for example simply with the addition of an excitation power supply and a resistor to create a voltage in the range of 0-5V that could be measure by the built-in ADCs on the E100-8Q.

 

Parameters to be measured

The following parameters were to be measured:

• Liquid Flow. We monitored UPW lateral Rotameter float positions with photoelectric switches to alert users of lateral flow changes

• Fan vibration. Monitored existing POR accelerometers on an active scrubber fan to alert users of an increasing over all fan vibration.

• Temperature and Humidity. Temperature and %RH were measured using 4-20mA instrumentation.

• Weight. We monitored polymer PoU (Point of Use) container weight and could alert users when the chemical level was low. In addition, a second scale monitored chemical containers in inventory and could notify users when the storage levels were low.

 

Data Transmission and Storage

Our client considered all data gathered by the system to be confidential and data collected would not be allowed off the corporate network. To facilitate this requirement we recommended installation of the ubiworx™ broker in the client’s private on-site data center. The IT department provided a virtual machine where we installed Debian 8 and the ubiworx™ server package. The ubiworx™ server package uses Apache, MySQL, PHP and the ubiworx™ broker to create a complete private cloud data and device management system.

By default, ubiworx™ encrypts all data sent to and from the broker using SSL with both client and server certificate authentication. This is optional though and for this deployment, the client opted to transmit data in the clear over MQTT since all wired networks used were private and secured by the client IT department. Wi-Fi network segments were encrypted using WPA2-AES.

 

Wi-Fi Networking

The client requested 802.11 Wi-Fi networking for communication with leaf nodes. Wi-Fi is not usually recommended for industrial environments but since this was a proof of concept, the client was more interested in learning if it is possible to deploy using Wi-Fi and if not what the impediments were.

Three gateways were configured as access points. These gateways were configured to provide private Wi-Fi networks for the leaf nodes and provide a wired connection back to the root server. The access point nodes were NAT enabled to hide leaf node IP numbers but no other firewall configuration was required.

The access point nodes used the same Supermicro gateway as the leaf nodes to maintain a single hardware SKU for all gateways. They also ran Windriver Linux and ubiworx™. Although the access point nodes did not have any sensors directly attached to them, it was still advantageous to run ubiworx™ as this allowed the client to manage the devices and monitor parameters such as Wi-Fi network signal strength, CPU usage and other indicators, as well as having the ability to remotely manage security settings and upgrade firmware as required.

 

 

 

System Architecture

Overview

A total of seven Supermicro gateways were deployed in a two-tier network configuration. Three were deployed as Wi-Fi access points (or “branch”) nodes and four were deployed as Wi-Fi client (or “leaf”) nodes. The broker (or “root”) node was a Linux VM provided by the client.

 

 

 

 

ubiworx™ Broker Deployment

The ubiworx™ broker collects data from all clients and provides a single web portal to manage an entire estate of gateways and sensors. The broker allows connected gateways to securely and reliably transmit data. By using a sequence number based handshake with each gateway, the broker ensures no data samples are ever missed due to unplanned or planned network outages.

The normal mode of operation for a ubiworx™ broker is to be deployed in the cloud. However it is also possible to deploy the ubiworx™ server locally on a

client’s own network. In this case, for data security reasons, the client elected to run the broker in their own private data center.

The broker was deployed on a virtual machine provided by the client which was running Debian Linux version 7 known as “wheezy”. Emutex staff took care of the initial installation and configuration of web and database services as well as the ubiworx™ broker application software. Once up and running, maintenance was taken over by the client IT department.

 

Sensor Management

All sensors were configured using ubiworx™ to have correct offset and scale factors to ensure meaningful readings. Sensors also had reporting intervals configured, typically of 30 seconds, using ubiworx™. High and Low level limits were configured for every sensor to allow alerts to be created when normal ranges were exceeded.

Where possible, fault detection was also configured. For example, the 4-20mA sensors generated a voltage input in the range 1-5V to an ADC that could sense in the range 0-5V. In this case, any reading below 1V was considered a fault and an alert was generated. The slight loss of resolution was deemed an acceptable tradeoff to gain the ability to detect faults.

 

 

 

 

Gateway Management

Each gateway was managed through the ubiworx™ server console interface. At a glance, it was easily confirmed that all gateways were online and functioning correctly. Alerts were generated whenever a gateway dropped offline through poor Wi-Fi or other reason. Other parameters such as free memory, CPU utilization and disk space were remotely monitored as well. Firmware upgrades could also be securely pushed to one or all gateways through the broker interface.

 

 

 

Alarm Management

As mentioned previously, any sensor reading outside the configured high and low level limits generated an alarm, as did communications and power failure conditions. These alarms appeared in the alarm console of the ubiworx™ broker web interface. The alarm action for each alarm was set to send an email to a distribution list. Alarm on and alarm off events were notified to the distribution list. This list included Emutex staff for system monitoring. It would also have been possible to set up text message or other alerts but it was felt email alerts were sufficient for this project.

 

SCADA Interface

The broker was required to have an interface to the corporate systems SCADA system. For simplicity, this interface was built around shared CSV files. Every 30 seconds, the broker would create files in a shared drive that contained timestamps and values for all connected sensors in the system.

The CS systems all run the Windows operating system, as is quite common in industrial environments. IoT gateways and other embedded systems more typically run Linux. This introduced some challenges around use of shared drives but through the use of SAMBA and some special access permissions on the shared folder, we were able to access a Windows share from a Linux gateway.

 

 

 

 

SCADA Data and Visualization

Data imported into the SCADA system was used to create mimics and trend graphs as detailed in this section.

 

Flow, temperature and humidity

 

 

Figure 2 shows a visualization of the instantaneous flow, temperature and humidity parameters.

 

 

 

This figure shows a trend graph of the temperature and humidity. It also clearly shows an outage where power was cut to the area being monitored due to maintenance works.

 

 

Vibration Monitoring

 

 

Figure 3 shows the fan vibration monitoring.

 

 

Polymer Weight

 

 

Figure 4 shows the polymer weights in production and in storage. To measure the polymer weight, a set of four load cells to measure the remaining weight of liquid used in the production process. This liquid was consumed slowly from small barrels which are periodically replaced.

Prior to deployment of the IoT monitoring system, each barrel was placed on a bathroom scales. An operator used to walk around the plant every morning and record the weight on each scale to determine the remaining amount of the liquid.

 

One of the planned benefits was to obviate the need to physically inspect the weighing scales to find the amount of liquid.

By utilizing connected gateways with load cells, an instantaneous value was always available online.

 

 

 

Results and Analysis

Sensor and Gateway Management

A key benefit of using ubiworx™ is of course the built in sensor and gateway management capability. This allows complete management of sensor configuration, scan interval, scaling, limits and alarms from a central web portal. In this project, the customer was highly impressed with this ability and wrote in their internal summary:

“ubiworx^™ proved to be an excellent tool to manage the gateway software and the gateway networks. ubiworx^™ has an easy to use graphical interface for adding, editing, and managing the gateways and can display basic data parallel with SCADA when required.”

This centralized management aspect is designed to enhance scalability and efficiency by collecting all management functionality in a single web portal. The client appreciated this and wrote in their summary:

“ubiworx^™ has significant advantages that would offer productivity and reliability gains for gateway additions, changes, sustaining, and network management and when dealing with large gateway systems.”

 

Choice of Gateway

The choice of the Supermicro E100-8Q IoT Gateway proved to be a good decision. The primary advantages of the E100-8Q are the low cost due to the Intel® Quark CPU and the integrated digital (GPIO) and anaog (ADC) I/O interfaces. This also reduced the cost by obviating the need for external I/O modules. Running Linux on the gateway also offered significant flexibility due to the wide range of tools and applications that are available.

At the time of developing this project, there were few options available in Intel® Atom class gateways that had integrated I/O. With the planned availability of the UP board from AAEON in May 2016, this option will open up (pardon the pun) and should be explored to increase the amount of processing that can be done at the edge and to allow the introduction of local display and other options.

 

Choice of Sensors

In general, the choice of passive sensors worked well and the devices chosen were cost effective, easy to install and reliable.

One recommendation however for future projects would be to use amplified load sensors for weight, preferably in the 0-5V range. The load cells used required amplifiers to be created which led to some problems with noise and offsets.

Similarly for IEPE accelerometers for measuring vibration, the cost advantage to the client of fabricating their own signal conditioners proved to be a false economy. Off the shelf Meggitt signal conditioners cost more, but are easier to install and have many advantages from a sustaining standpoint.

 

Cost-effective stock level maintenance

By setting low level limits in ubiworx™, e-mail alarms were generated whenever the remaining quantity dropped below the reorder threshold. These emails alerted operators to change the barrels.

In addition to this planned benefit, it quickly became apparent there were a number of other benefits to automated monitoring. Firstly, as the quantity was available in a database minute-by-minute, it was now possible to calculate consumption rates for each of the ingredients and correlate these to production times. It can now be automatically verified that actual consumption rates are matching predicted consumption rates and it can be predicted when a change is required.

Secondly the low level threshold alarm duration can be measured to ensure alarms are dealt with and ingredients are replenished in a timely manner. The low level alarms can also be time-correlated with any downtime in production to identify possible causes of downtime.

 

Wi-Fi

The use of Wi-Fi in a production environment was known to carry some risk factors. 2.4GHz signals do not have good penetration of concrete and steel structures typically found in a factory environment. Also, the ISM band used is prone to interference from many sources. This was understood at the start of the project but it was decided that it would be useful to evaluate it as an option.

As expected, Wi-Fi signal strength was variable and drop-outs were experienced. A sensor was added to ubiworx™ to report on Wi-Fi signal strength which proved to be invaluable for tracking signal strength during the day. The ubiworx™ store-and-forward feature worked correctly though and ensured no data loss occurred.

 

 

 

Conclusions and Next Steps

There were several key learnings for us and for our client from this project and there are numerous directions in which we plan to take this work.

Our first learning is that standard 802.11 Wi-Fi is inadequate for an industrial environment. Wired networking is preferred if at all possible but if not then the use of Zigbee or perhaps even Lora or Sigfox would be preferred to Wi-Fi.

We did of course also verify that tore and forward is an invaluable feature in ubiworx™. Without the ability to replay missed samples, data would certainly have been lost.

Export to SCADA using shared files is problematic. Windows and Linux sharing a file system is another complication that should be avoided. OPC-UA was seen as an unnecessary complication at the start of the project, but would in fact have saved a lot of time in the long run. In future, gateways will use OPC-UA to connect to SCADA systems.

 

Some of the benefits of the system were immediately obvious and were intended from the beginning but some only became apparent as the system became used in daily operation. In addition, some benefits may not be seen for some time to come. As with all IoT and big data systems, it is important to record all data for later data mining even if the reason is not currently known. Future analysis or correlation detections may be able to use the data gathered in ways that cannot currently be anticipated.

Overall, our client was happy to announce that this trial passed all of the original test parameters we established for data delivery on flow meters, temperature/humidity sensors, fan vibration accelerometers, point of use chemical weight scales, and also passed a 21 day Wi-Fi test run. There were some challenges with the gateways hardware, Wi-Fi, and system architecture along the way, but all issues were resolved and resulted in manufacture hardware design changes, Wi-Fi configuration recommendations for Supermicro gateways, and about 39 other lessons learned that will help improve future installs.