Tag: Ubuntu

Advantech AIR-020X Review

Normally, I am not getting review units. This is due to the fact that I am only hosting this small weblog, along some conference talks - and most companies would probably be better off to send their units along someone with a reach of Linus Tech Tips, or similar.

On the other hand - when I get the possibility to do a review, it can be a bit worrisome for the companies as well, as I am a very honest person. I have been working in tech for some time now and had the honor to build stuff which went to space - and came back to tell the tale. I know what I want in a unit - and what could be a problem.

With this out of the way, I was one lucky winner of the Advantech Edge AI Challenge 2022 and got an AIR-020X-S9A1 unit at no charge to be able to realize my labSentinel 2 project. By doing this project I learned a bit about the box and thought it would not be a bad idea to share my ideas with the readers of my blog - and also Advantech, so that they can improve upon their product. This review is not paid for, reflects my own thoughts and I got the mentioned unit for my project - the review was not a part of that deal. With that out of the way, lets get started.

The hardware

The AIR-020X comes very well packaged - having its own foam jacket which will save it from all but the most horrible abuse from postal services. Not that it would matter: The roughly 14 cm x 12 cm x 4,5 cm compact unit weighs in at nearly 850 gr and is built sturdy and robust - like a tank:

The most obvious part of the unit is its heatsink, which it does put to good use - but more on that topic later. Along with the computer itself comes a chinese printed starting guide and a short USB A to Micro B cable, which will be needed to factory reset and reflash the unit.

All in all, the AIR-020X is an impressive unit, including an Nvidia Jetson Xavier NX module with 8 GB RAM, 16 GB onboard eMMC, 128 GB M.2 Flash, 2x RS232/422/485, 1x CANbus, 1xDIO ("GPIO"), 2x 1 Gbit ethernet, 1x Fullsize mPCIe with nano SIM holder, 1x 4k HDMI Output, 2x USB 3.0 Type A, 1x USB Type C. The unit is powered by a 12-24 V DC power supply, which is an optional accessory.

Being an industrial unit, it uses an industrial type connector for power, which is an HT5.08 2 pole type:

As this connector is also not part of the base package and the USB C connector does not accept power delivery (and neither works in Display Port Mode) - it becomes a bit harder to power up the unit after receiving it. Finding a usable power supply within the sizable voltage range of 12 - 24 V (e.g. from an old Laptop) is fairly easy, but without the connector - it becomes a dead end until the next delivery is there. It would be useful to at least include one connector with the base unit. The usb cable is a nice addition, but could be left out (even though its very high quality) - along with the chinese manual. This could be replaced with a small card with direct links to the english and chinese PDF versions of the manual.

Opening up the unit reveals the internals - but not without a fight:

The used screws are perfectly fixed to the structure by using blue loctite - a touch I cannot recommend enough for the vibration resistance of the overall unit - but the screws themselves are made from extremely soft metal, so that - using the correct screwdriver - I stripped nearly all screws and had really issues removing all of them. Somehow this problem seems to exist for all the external black screws, the internal silver ones were of a lot higher quality. In my case I fixed the issue by replacing the screws with new ones and never had an issue anymore with them.

The internal structure is very well laid out, raising the M.2 drive onto a pole to keep it a bit further from the heat source / Xavier NX module which is just sitting on the other side of the PCB and directly sandwiches with the big heatsink.

Very welcome are also the addition of the two Raspberry Pi Style Camera connectors, although they are a bit hidden by the serial console cables. I understand that the unit should be as closed as possible for the use in factories, but I would have loved to see two small slits (possibly even with some IP/EMC gaskets to allow for protective shielding of those entry points) so that cameras on the outside of the case can be easily attached.

The mPCIe slot gives the system an additional expansion slot for e.G. UMTS or LoRaWAN modules and also the internal CR2032 cell for the RTC is a small but valuable detail.

The AIR-020X has some mounting points available on both system sides for additional wall mounting rails. Looking at the mounting points and the obvious use of the AIR-020 series in lab and factory settings, the inclusion of a DIN rail mount as available accessory could prove very useful to directly mount this small computer into an electrical cabinet.

The software

Booting up the system greats one with a very familiar picture: Ubuntu 18.04 is running on the machine in form of a tailored version of Nvidia Jetpack. This version by Advantech is only using the eMMC of the Xavier NX module to start the bootloader, but the actual data is kept on the M.2. This is a great idea for the longevity of the eMMC on the (currently hard to find) Xavier NX module - but comes with the drawback of additional needed customization other than "only" the PCB, included hardware, drivers and other changes made by Advantech in comparision to an Nvidia Developerboard for the same module.

This is a problem I also learned the hard way: I realized that the board was delivered with L4T 32.5.2 - not the current 32.7.x (JetPack 4.6.1) - so I updated this by hand. Just to have the board bootloop. This was the moment I took a closer look to the online presence of Advantech and the manual - just to learn that the recovery process was neither described, nor was the download of the image available. I got the needed recovery file as well as the documentation (which also included vital information on how to use the DIO (GPIO), RS422 and CANbus interface) and as able to restore the board to working order. Obviously there were multiple problems with this: First, the online available manual should contain all needed information regardings settings, ports, recovery, etc - secondly, the current (and maybe even last) images also need to be available online on their website - with checksums to be able to deploy these images safetly.

I also voiced my concerns regarding the high impact security issues / CVEs found in 32.5.2 - which would make the use of AIR-020 series an absolute liability in a production environment. I am glad to report that Advantech reacted to these concerns with providing a beta version of a new JetPack 4.6.1 Image. A short time afterwards, Advantech did add some information to their wiki:

On the download page you can find the AIR020A2AIM20UIV00004 entry for the Jetson NX JetPack 4.6.1 from 2022-07-20. This links to a Dropbox folder containing a the latest image (AIR020A2AIM20UIV00004_194.tar.gz / 2022-09-16).

With this latest image I was able to upgrade the AIR-020X to JetPack 4.6.1 and even do and apt upgrade to upgrade to L4T 32.7.2, at the time the latest L4T. However, this did not go as planed: After doing the upgrade and rebooting the device, it got caught in a bootloop. This bootloop kept on repeating for about 10 minutes until the device mysteriously started then working and came back on without issues. Obviously this would not be a graceful upgrade and did instill some concerns why this was a reproducible issue.

I am glad to report that Advantech has provided the latest image - which will eliminate several security issues. However, the changes needed in the manual as well as the provision of the recovery images (now via Dropbox?) and the secure provision of security updates to the unit remain. Maybe Advantech would think about starting to use balena.io to handle these issues?

Verdict

The Advantech AIR-020X is an extremely capable unit in a small form factor, sturdy built and highly reliable. Even with the latest JetPack 4.6.1 and abuse of the formerly not available 20 Watt mode I could not get this unit to heat up too much in my testing with labSentinel 2. There is still enough headroom available to use it in any kind of environment, which makes it a perfect choice for labs and factories - if Advantech can tackle the presented issues. Especially the ones regarding timely and secure availability of security patches and software updates. This also means availability of these images, fast adaption after release of official Nvidia updates and all needed documentation in one manual for public download. With these exceptions and some small kinks, Advantech is so close to building the perfect unit for their envisioned use case. I really hope they can close that last (security/software/manual) gap to an otherwise nearly perfect hardware - and with that create an recommendable product.

Edit: balenaOS

I got balenaOS working on the device - see here.

Ubuntu 20.04 Update bricked KVM Virt

I updated an older Ubuntu 18.04 LTS system to the latest LTS and had (among other things) Docker and KVM installed. KVM is actually quite nice if you "just need" a small VM (pfsense ;)). I actually prefer Proxmox and ESXi, but hey, the right tool for the right job.

After the upgrade to 20.04, kvm did not work anymore and I got a lot of lvm2 errors during apt update / apt upgrade sessions, so a short google later I found this. I was a bit nervous, but the fix did neither hurt my kvm nor my Docker instances

sudo apt purge lvm2 && sudo apt install lvm2

(The fix is deleting and reinstalling lvm2)

After reinstalling lvm2, I could successfully execute a virsh list and got my list of running KVM machines back:

 Id   Name      State
-------------------------
 1    pfsense   running

[Dell] T30 Intel AMT Blank Screen on Ubuntu Fix

The Dell T30 is an awesome little Homeserver, packing a punch with the Xeon E3-1225 V5 - and being affordable at about 399 €. It also comes with Intels Active Management Technology / AMT which is an extension of the horrible Intel Mangement Engine (which was all over the place months ago when some genius figured out how to stop that Man-in-the-Middle-always-on chip with some simple commands) - but quite useful - nonetheless. The good thing about this, is that it acts like an DRAC (Dell) / ILOM (Sun) / IPMI (Supermicro) card - so it is an KVM (Keyboard Video Mouse, not the virtualization thingy this time, sorry ;)) extension which allows you to control the server via network as if you were plugged in directly.

There is an awesome guide from Christian on goNeuland, written in German on howto setup that thing without the need to buy VNC Viewer Plus.

However, my Ubuntu instance came in as blank screen after successfully connecting to the system. In the end, that turned out to be that way, as Ubuntu decided to deactivate the graphics unit - due to no monitor being attached.

Different solutions were talked about here, here and here.

In my case, following helped:

1.) Open your grub, i.e. sudo vi /etc/default/grub file

2.) Add nomodeset to your GRUB_CMDLINE_LINUX_DEFAULT line, so that it would read i.e. GRUB_CMDLINE_LINUX_DEFAULT="reboot=force bootdegraded=true nomodeset" (your commands will vary!)

3.) Save and close the file

4.) Update grub via sudo update-grub

And after a quick reboot, everything worked out :)!

Upgrade WSL (Windows Subsystem for Linux) on Windows 10

I had installed WSL (Windows Subsystem for Linux) a long time ago to gain access to Ubuntu 14.04 LTS directly from my Windows 10 Desktop. However, as time passes, Software grows old. Upgrading the Ubuntu Subsystem via apt-get update / do-release-upgrade should work, but that could have some nasty sideeffects, considering that the 14.04 LTS WSL release had been a beta test - so, a reinstall should be better.

Luckily, TechRepublic got this covered. Just open a CMD and run:

lxrun /uninstall /full /y

to uninstall the current WSL version.

Afterwards, try

lxrun /install

to reinstall it. With this "reinstall", Ubuntu 16.04 LTS will be installed.

Nonetheless, I recommend a nice

sudo apt-get update
sudo apt-get dist-upgrade

afterwards in your BASH session to get the WSL to the latest version ;).

 

[Ubuntu] PERC6/i on Ubuntu 16.04 LTS

To use the PERC6/i i.e. the

03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)

on Ubuntu, megacli is the best tool - but rarely available due to the demise of LSI Logic. Good thing that the guys from https://hwraid.le-vert.net put together a nice repo to host the latest RAID files. And yes, for everyone that does not like the idea of including a foreign repo - sorry to disappoint here :/.

# Add GPG signatures
wget -O - https://hwraid.le-vert.net/debian/hwraid.le-vert.net.gpg.key | sudo apt-key add -

# Add Package Repo
echo "deb http://hwraid.le-vert.net/ubuntu xenial main" | sudo tee -a /etc/apt/sources.list.d/hwraid.list

# Upgrade and Install
sudo apt-get update
sudo apt-get install megacli

After that, megacli is installed and can be used:

# Basic Commands
# Info Controller
sudo megacli -AdpAllInfo -aAll
sudo megacli -CfgDsply -aALL

# Info Virtuelles Laufwerk
sudo megacli -LDInfo -Lall -aALL

# Info Battery
sudo megacli -AdpBbuCmd -aALL

I picked out the most important infos for me and wrote this little script

#!/bin/bash

echo "Some Infos are commeted out in this script to not overwhel the user ;)"

#echo "----------------------- RAID Controller"
#sudo megacli -AdpAllInfo -aAll

#echo "----------------------- RAID Controller Config"
#sudo megacli -CfgDsply -aALL

echo "----------------------- RAID Battery"
#sudo megacli -AdpBbuCmd -aALL
sudo megacli -AdpBbuCmd -aALL | grep "Battery State:"
sudo megacli -AdpBbuCmd -aALL | grep "Charger Status:"
sudo megacli -AdpBbuCmd -aALL | grep "Relative State of Charge:"
sudo megacli -AdpBbuCmd -aALL | grep "Next Learn time:"

echo "----------------------- RAID Virtual Drive"
#sudo megacli -LDInfo -Lall -aALL
sudo megacli -LDInfo -Lall -aALL | grep "State"

echo "----------------------- RAID Harddrive Status"
sudo megacli -CfgDsply -aAll | grep "Drive has flagged a S.M.A.R.T alert"

 

Additional infos can be found on:

http://erikimh.com/megacli-cheatsheet/

https://www.thomas-krenn.com/de/wiki/MegaRAID_Controller_mit_MegaCLI_verwalten

Reboot Ubuntu if init daemon fails / systemd died

I got following error on an really badly "injured" Ubuntu machine:

Failed to start reboot.target: Failed to activate service 'org.freedesktop.systemd1': timed out
See system logs and 'systemctl status reboot.target' for details.
Failed to talk to init daemon.

Rebooting was not possible anymore, due to the beauty of 2933 zombie processes - slowing down the machine to a crawl.

However, this neat issue provided the answer to STILL get it to reboot:

# Reboot
sudo systemctl --force --force reboot
# Shutdown
sudo systemctl --force --force poweroff

"Cool", as Gregory House would say 😉

[Ubuntu] Use Molly-Guard to stop shooting your own leg

If you're working on some dozens of linux servers (or even more than 100,.. as in my case), you end up doing administration via SSH - which is the way to go. And chances are, that you'll get dozens of SSH connections open in dozens of tabs and you did some updates on some of those servers and want to restart this thing with a quick sudo reboot now...
I won't lie if I say, it happend more than once that I accidentally rebooted the wrong server - at least that was the case more than a year ago.
For the last year, since I have been using Molly-Guard - that did not happen once. Why? Because Molly-Guard does stop the reboot command if it detects that you're issuing it from an SSH console - and asks for the server name. If you're entering it correctly - it will reboot. If you're in a frenzy, doing your "sudo reboot now" and enter name serverB while you're on serverA - yep, Molly-Guard will stop you from shooting yourself in the leg. Neat, ain't it?

Oh - and the best part? Ease of use: sudo apt-get install molly-guard
Thats it, you're set, bye.

Nope. Really. No configuration needed. Just install that baby and be safe :)!

[Ubuntu] Letsencrypt with Apache and Freeradius

This little tutorial describes how to use Letsencrypt with Apache, Freeradius and Auto-Renewal of the Certificates.

#Install Letsencrypt
sudo apt-get update
sudo apt-get install git
cd /opt
sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
cd /opt/letsencrypt

#Become root
sudo su

#"Order" certificates (replace SERVERDOMAIN.COM with the DNS of your Server!)
./letsencrypt-auto --apache -d SERVERDOMAIN.COM --rsa-key-size 4096
Enter Contact Mail: mail@SERVERDOMAIN.COM
Configuration Type: Secure #is best, as it does redirect insecure http to https)

#Read PATH variable
echo $PATH

#Cronjob for certificate renewal
#you should under all circumstances replace the string following PATH= with your own, as read with the command above.
#Seperate with ; from the rest of the command like shown in the example
crontab -e

#letsencrypt
30 2 * * 1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games;/opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log
35 2 * * 1 /etc/init.d/freeradius restart
35 2 * * 1 /etc/init.d/apache2 restart

#Configure Freeradius
cp -r /etc/freeradius/certs/ /etc/freeradius/certs_bkp
rm /etc/freeradius/certs/*.pem
cp /etc/freeradius/eap.conf /etc/freeradius/eap.conf_bkp

vi /etc/freeradius/eap.conf

#certdir = ${confdir}/certs
#cadir = ${confdir}/certs
certdir = /etc/letsencrypt/live/SERVERDOMAIN.COM
cadir = /etc/letsencrypt/live/SERVERDOMAIN.COM
#dh_file = ${certdir}/dh
dh_file = ${confdir}/certs/dh
#private_key_password = whatever
private_key_file = ${certdir}/privkey.pem
certificate_file = ${certdir}/cert.pem
CA_file = ${cadir}/fullchain.pem

#Configure access rights on /etc/letsencrypt
cd /etc/letsencrypt/
chgrp -R ssl-cert archive csr keys live options-ssl-apache.conf renewal # set group of cert/key dirs to ssl-cert
find . -type d -exec chmod g+xs {} \; # directories executable and setguid (set group ssl-cert for new files/dirs)
find . -type f -exec chmod g+r {} \; # files readable

#Restart Freeradius
service freeradius stop
service freeradius start

Additional infos: https://www.digitalocean.com/community/tutorials/how-to-secure-apache-with-let-s-encrypt-on-ubuntu-14-04

[Ubuntu] Install Docker

This is a short guide to install the recent Docker Version from the official ppa on Ubuntu 14.04 LTS - along with some other great tools like docker-compose.
Please bear in mind, that Docker needs an 64 Bit System to work with :)! So no i686 plattforms from here on.

# Add Docker Key
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
# Add Repo
echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" | sudo tee /etc/apt/sources.list.d/docker.list
# Update and Install
sudo apt-get update
# Install Recommended Package
sudo apt-get install linux-image-extra-$(uname -r)
# Install Docker itself
sudo apt-get install docker-engine
# Useradd - so you can use docker with your own user without sudo
sudo usermod -aG docker ${USER}
# Install pip
sudo apt-get install python-pip
# Install docker-compose
sudo pip install docker-compose
# Test Docker Install
sudo docker run hello-world
# After an additional reboot, you will be able to use docker with your own user (also recommend because of the the new linux-image-extra 🙂