Custom Application Troubleshooting With Cisco IOx

Cisco IOx provides you with a platform to host your microservices at the networks edge, close to your data originator. Microservices allow you to process your data, apply filtering, aggregation, densification, quality checks and much more!

What to keep in mind?

Are Linux based systems and Docker applications old friends of yours?  If so, you probably already know about some challenges of creating Docker images. With IOx, creating your own custom application is a well-defined process. There are some subsequent steps during which you will create an application docker image, define operational parameters, and how many resources to provide to your application.

Consider the following as a checklist to keep in mind, when creating an IOx docker-based application image. Maybe some of the following challenges sound familiar to you.

Potential challenges when creating Docker images

1. Your final Docker image became too large?

Avoid using overly large docker-base images. Try to use a minimal one! If you can manage to use “scratch” or “alpine” the lifecycle-management of your docker image during deployment and redeployment will cause less overhead, waiting time, and allocate less bandwidth and storage on the device. Check the following quick comparison of different base image sizes. Keep in mind: your application framework, libraries, binaries, tools, and interpreters will be added on top of this size.

Note: Good candidates are most likely “ubuntu” with ~73MB, or even better “alpine” with only ~6MB base image size. If your application has minimal external dependencies, the optimal choice might be to build your application image from the smallest possible base image “scratch,” which is even smaller than alpine. The additional effort for using smaller images vs. more convenient and larger base image, is well invested time!

2. Did you run out of storage when deploying an application package?
Keep in mind that you cannot claim all the available storage to IOx for your application. What you will deploy as an application package is a compressed archive containing the docker filesystem layers. Your device will already allocate storage to receive, store, and uncompress your image. That in turn means that effectively your image will cause storage to be allocated approximately 2.5 times of its own size!

3. Is your application storing data frequently?
If the application you want to run on IOx is using storage to persist information, please keep in mind to equip the device with an SSD expansion module to support your plan. This is to provide additional capacity and to avoid the main system flash from wearing out.

See:

4. Does your application create large container logfiles?
Keep in mind that stdout and stderr are being logged. You should only log important messages in the production version of your application using the application logfiles. You should also consider using a log rotation mechanism if the logs are getting large.

5. Are you unsure about errors during the operation of your application?
Make sure your application is logging information about its operational wellbeing. The standard paths for logging are device dependent. On IR8x9 you would use “/data/logs” and on IR18xx and IR1101 you would use “/iox_data/logs” to store your own logfiles.

To refer to the different locations without hardcoding it, please check the link below on environmental variables, which will abstract the different paths for you!

Think about time and size-based log rotation to prevent the filesystem from filling up with application logs. If you stick to the mentioned log locations, you will be able to access the logfiles from any of the available control planes – IoT Operations Dashboard, Local Manager, ioxclient, and FND. Your application log is obviously the go-to point if you experience issues during operation with your application!

See:

6. Your device logfile protocols “not found” errors during startup?
Make sure you comply to the IOx default paths for application files and logs. For your application files they are for instance “/iox_data/appdata” for IR1101/IR1800 and “/data/appdata” for IR8x9, whereas for your logs it is “/iox_data/logs” and “/data/logs”.

An example message from device logs:

{"log":"/bin/sh: /start.sh: not foundn","stream":"stderr","time":"2022-04-28T14:01:12.945697208Z"} 

7. Is your device running out of resources while deploying multiple applications?
If you are unsure how much system resources to provide to your application to fulfill its task without overprovisioning it, you could make it run exclusively, and monitor its real usage of CPU and memory inside the container while being under realistic operational load. You can then later adjust the resources provisioned to your application to match your observation.

See:

8. Are you unable to access ports that your application exposes?
If you intend to expose ports for communication, make sure that your device configuration will use for instance NAT and proper port exposure on the corresponding port of your Docker container instance. Evaluate if assigning a fixed IPv4 to your container eases debugging an error situation. Do not forget to check the port exposure in your Dockerfile, as well as your package.yaml. In addition, the package.yaml will also define the applications access to services like GPS, accelerometer, serial port and more.

See:

9. Does your application refuse to start on the target hardware?
The architecture varies between different Cisco IoT devices. For example, IR1101/IR18xx provides an ARM64 based architecture, whereas IR8x9 does provide a x86_64 based hardware architecture. Make sure to have your base image selected and your software compiled or cross-compiled for the correct target architecture, or else it will fail to install or start up. Choosing the wrong target hardware architecture for your device will either result in not being able to upload/activate your image, or the startup will fail. The latter will cause a corresponding log entry in your device logs to be written.

See:

10. Are you unsure about the operational state of your application?
Leverage health-monitoring for your application! The built-in health-monitoring provides you with the possibility to have your application frequently checked for correct operations. The result of the health check is automatically provisioned to the control plane of your choice.

See:

11. Does your application fail to start and there is no application log created?
If you cannot find your application log (see 8.), it might be due to a misconfiguration of the device itself. The go-to resource is the operational logging of your device itself. Depending on your setup, you will find the logs via IoT Operations Dashboard, LocalManager, FND, or ioxclient.

12. Does your application start and immediately stop again?
Double check if your Dockerfile specifies a blocking command using for example CMD or The process you specify needs to run without interruption, or your IOx application will stop. You might need to catch exceptions if you use a script to start processes.

See:

13. Does your application start, run for a while, and exit while being under load?
Check your applications memory and file descriptor consumption. It might leak on either of them. If your application does not properly free memory and/or filedescriptors, it will eventually be forcefully ended by the kernel. Swapping memory is not an option! To confirm, if you are being impacted by this problem, plot the consumption on memory and filedescriptors over time.

The following example illustrates the ever-increasing allocation on VSS/RES memory of a process. The lack of saturation suggests a memory leak, that eventually will lead to this process being forcefully ended.

14. Does activating your IOx application fail with an “ID syntax error”?
Make sure you use your docker image name:tag during export, or you might experience issues with the application naming in your Dockerfile, manifest, or package.yaml when activating it.

Example from LocalManager:

15. Is your final application image not compliant during security scans, and/or building time takes too much time?
Try to harden your image by removing all unnecessary binaries, libraries, and caches. This will not only reduce the footprint on storage but help to reduce security threads for software packages and libraries you must patch and monitor for updates. This also includes caches the system built during image assembly. The easiest approach is to leverage Docker multi-stage builds. You would only copy the required things to the last build stage and remove everything unnecessary, like for instance system binaries your production application image does not need. Multi-stage builds allow you to create debug images and production images at the same time.

The above example illustrates the difference of “intermediate-stage” to “final-stage” regarding their size. The result docker image is functionally identical in both cases. Both originate from different stages of the same image build process.

See:

16. Do you know if any of the components you used in your application image is being affected by security issues?
Consider checking your Docker image security compliance with available toolkits such as “Docker Bench.”

17. Do you want to shortcut the testing of your application and only deploy previously tested application images?
To save time and effort test your docker image in your local environment before creating an IOx application. Also consider ramping up a DevNet sandbox for testing and learning!

See:

Where to go from here?

Join our daily livestream from the DevNet Zone during Cisco Live!

Stay Informed!
Sign up for the DevNet Zone Cisco Live Email News and be the first to know about special sessions and surprises whether you are attending in person or will engage with us online.

We’d love to hear what you think. Ask a question or leave a comment below.
And stay connected with Cisco DevNet on social!

LinkedIn | Twitter @CiscoDevNet | Facebook | YouTube Channel

Share: