Testing IoT for interoperability, security and reliability

What needs to be tested?

Networking is complex and home networks, in particular, rely on a number of different protocols and applications that smart products require or interact with. Some of these underlying technologies and protocols, like Wi-Fi and IPv6, were built with many options and scenarios that make interoperability testing of each component critical.

Connectivity and core protocol behavior

Though IP networking is the standard that makes our networks operate, it is not simple. Devices connecting to the network need to be able to handle basic physical and medium access control layer connectivity, address assignment and address resolution. Even the simplest device needs to use Dynamic Host Configuration Protocol (DHCP) and DNS — the latter of which is the source of many user experience problems and security vulnerabilities.

Additionally, because most gateways make use of IPv6 in some capacity, smart devices need to know that their IPv6 implementation will work and work well. When testing DHCPv6 and stateless address auto configuration, the assignment and communication of link-local, unique local and global unicast addresses, and even handling DNS over IPv6, are critically important for smart devices that need to interoperate and may have intermittent connections or sleep modes.

Wi-Fi interoperability

Consumers purchasing smart home products expect them to work within their home network. Wi-Fi is the standard for in-home networking but has a significant number of options and features for security, backward compatibility, operational efficiency and regional compliance.

Developers need to test smart devices across all modes of operation, over different frequency bands and channels, with different country codes and across all security modes. Backward compatibility is the biggest challenge to Wi-Fi interoperability. Ensuring that smart devices will work anywhere is critical to the end-user experience.

Stability and performance

In addition to validating that a smart device will connect and operate in various network environments, developers should stress their products’ performance over long periods of time and ensure that they can recover from reboots or sleep states. Smart devices may connect and reconnect to the network frequently, and memory leaks and memory fragmentation can cause performance issues or disable a device altogether.

Security

Obviously, security is the most talked about topic in IoT and smart home product media. It is one of the most cited reasons why consumers are nervous about purchasing and using these products. Getting a basic understanding of how a smart device appears on a network and how it can be accessed is a necessary first step in developing a secure product.

Home IoT devices are also mostly dependent on connections to cloud services for their applications. Monitoring the traffic that smart devices produce when connected to the internet and analyzing that traffic for insecure behavior is critical. Even the basics, like using HTTPS and implementing Transport Layer Security correctly, can be missed on any device.

What can be done to improve smart home tech quality?

Developers and integrators are in a tough position when it comes to these network technology issues. Because they are under pressure to deliver quality solutions quickly, they often expect network interoperability to come for free with their embedded Linux implementation. But even when using open source code, integration issues arise.

Realizing this, there are a number of things that developers can do to make their lives easier, and ultimately, to build better products for consumers, including these three steps:

Automate testing

Many IoT creators rely on manually validating connectivity, stability and performance. This is very labor-intensive and time-consuming. It’s also difficult to keep track of results and stay consistent throughout the entire process.

Use test automation tools that can operate without human interaction as much as possible. Include the ability to remotely restart or power-cycle products during testing. In addition to saving significant time and creating consistency with the test process, automation that can run independently makes it possible to perform intensive stability testing over long periods of time.

Supplement expertise

Companies creating their own test solutions, if they are doing so at all, often rely on their developers to code their own test cases and test software. This is even more labor-intensive than manually performing tests, and most networking protocols require specialized knowledge to implement send and expect behavior in a way that will improve product interoperability.

Find test cases that have already been built by experts and especially those that focus on these complex protocols and are frequently updated to stay on top of the latest technology.

Pay attention to the little things

Testing protocol interoperability and stability over time will reveal issues that would not arise until devices and applications are long in use by an end user. Subtle problems like memory leaks can threaten device stability or freeze Wi-Fi connectivity. Behavior that would be normal on its own can put a device in an insecure state with open network ports or worse when exercising the system.

Ultimately, a focus on the connectivity and interoperability of IoT devices through repeatable, automated testing gives developers and integrators greater confidence in their products and the services they support. That means happier consumers and a more successful industry as a whole.

About the author

Jason Walls is director of technical marketing at QA Cafe, a team of developers of quality test and analysis solutions for networks and networking products. A protocol geek at heart, he’s been involved with computer networking technologies for over 20 years and helps drive industry standards and open source projects for broadband, enterprise and consumer networks.