NFS Benchmarking RPi 3B, RPi 3B+ & BPi

3-14, PiDay. Sadly the day Stephen Hawking passed away - and the day the new Raspberry Pi 3B Plus was unveiled.
Some days later, I got a small package:

With that small thing at hand, I was looking through the list of changes:

  • Broadcom BCM2837B0, Cortex-A53 (ARMv8) 64-bit SoC @ 1.4GHz
  • 2.4GHz and 5GHz IEEE 802.11.b/g/n/ac wireless LAN, Bluetooth 4.2, BLE
  • Gigabit Ethernet over USB 2.0 (maximum throughput 300 Mbps)
  • Power-over-Ethernet (PoE) support (requires separate PoE HAT)

Ok, CPU bumped from 1.2 to 1.4 GHz, new WiFi with 802.11ac (and 5 GHz support), BLE 4.2, PoE "support" - and Gigabit over USB 2.0.
Well, not too bad.
One of the biggest downsides of the RPi - since its release - was the design choice to add Ethernet over USB 2.0 - while sharing the one and only USB 2.0 lane of the BCM with all the other USB sockets of the RPi. While this choice leads to lower production costs, it also limits the speed of Ethernet / USB especially in case where people tried to implement NAS or File sharing applications with an added USB Drive over Ethernet. To see how much the new design improved that old problem, I decided to Benchmark an "old" RPi 3B with the latest RPi 3B+ in the discipline of NFS.

Test setup:

  • Raspbian Stretch Lite (March 2018 / 2018-03-13 / Kernel 4.9)
  • WD PiDrive 314 GB as HDD via USB
  • Gigabit Ethernet Switch
  • Benchmarking system: Windows 10 with MS NFS Client, Crystal Disk Mark 4.0.3 and Gigabit Ethernet
  • NFS Exports Options: (rw,sync,no_subtree_check,all_squash,anonuid=65534,anongid=65534)

The benchmark started with the old RPi 3B and no surprises:

The RPi 3B is lacking throughput at its USB 2.0 lane.

Next up, was the new RPi 3B+:


And - yes - the new design really showed some improvement, to say the least.

However, I was wondering: What would happen if I added an SBC with a real USB 2.0 connection and real Gigabit Ethernet to the test?
Well - I did that. Welcome the now 4 years old Banana Pi 1:

We are talking here about a 1 GHz Dual Core A20 processor with 1 GB RAM which is - i need to repeat - 4 years old - releases just 2 months before the Raspberry Pi 1 B+ came out - I direct competitor of the RPi 1st Generation. But, what happens if we use the SATA port - which is also included on the BPi? With... lets say... some Samsung 850 EVO 250 GB SSD - which I had laying around? 😉

And what happens if I plug that in via USB?

Ok. Thats a bit confusing, to say the least. Could be that the SATA port is actually just coupled via USB to the BPi - however, I saw benchmarks on which the SATA port performed a lot better on the BPi. For this benchmark, I used ARMBIAN:

ARMBIAN Debian Server mainline Kernel
Armbian_5.38_Bananapi_Debian_stretch_next_4.14.14
Welcome to ARMBIAN 5.38 stable Debian GNU/Linux 9 (stretch) 4.14.18-sunxi

So, bundling an SSD with some old SBC board will not give you the needed freedom to just drop buying expensive storage for your datacenters and replacing them with small boards - sadly ;). However, we could see that the new RPi 3B+ improved a lot in one of its weak points - but still gets beaten by an very old - and still very feisty rival.

[VMWare] Deploy vCenter Server Appliance to ESXi 6.5

To deploy the vCenter Server Appliance (vcsa) to an ESXi 6.5 Server, you need to grab the latest vCenter Server for Linux from http://vmware.com/go/evaluate-vsphere-en - at the moment, thats the Version 6.5.0U1d from 19.12.2017 - which equals to the file VMware-VCSA-all-6.5.0-7312210.iso. After downloading that DVD Image, just burn it to a DVD or unpack it. You can then go on with Deploying it via GUI - or via CLI.

For deploying it via GUI, just start vcsa-ui-installer\win32\installer.exe - if you're on a Windows PC - or execute the installer / Installer.app files in the lin64 or mac folders.

If you want to deploy it via CLI using a template, you should navigate to vcsa-cli-installer\win32. Within that folder, you will find the vcsa-deploy.exe - that is the one we are going to use.

Now, we need to provide a template for deploying our vcsa:

{
    "__version": "2.3.0",
    "new.vcsa": {
        "esxi": {
            "hostname": "ESXI_IP",
            "username": "ESXI_USER",
            "password": "ESXI_PASSWORD",
            "deployment.network": "ESXI_NETWORK, i.e. VM Network",
            "datastore": "ESXI_DATASTORE"
        },
        "appliance": {
            "thin.disk.mode": true,
            "deployment.option": "tiny",
            "name": "NAME_OF_THE_VCSA_APPLIANCE"
        },
        "network": {
            "ip.family": "ipv4",
            "mode": "static",
            "ip": "IP_OF_THE_VCSA",
            "dns.servers": "DNS_SERVER_OF_THE_VCSA",
            "prefix": "NETMASK_OF_THE_VCSA",
            "gateway": "GATEWAY_OF_THE_VCSA",
            "system.name": "HOSTNAME_OF_THE_VCSA"
        },
        "os": {
            "password": "PASSWORD_OF_THE_VCSA_ROOT_ACCOUNT",
            "ntp.servers": "0.pool.ntp.org",
            "ssh.enable": true
        },
        "sso": {
            "password": "SSO_FOR_VCSA_PASSWORD",
            "domain-name": "DOMAIN_NAME_OF_VCSA_AD - could be vsphere.local",
            "site-name": "DEMOSITE"
        }
    },
    "ceip": {
        "description": {
            "__comments": [
                [
                    "This is a demo template with disabled CEIP / Customer Telemetry"
                ]
            ]
        },
        "settings": {
            "ceip.enabled": false
        }
    }
}

Just fill in the needed variables and save it in the root folder of your installer, i.e. as install_template.json. Then run the installer (in Windows, that means to execute following CMD): vcsa-deploy.exe install install_template.json --accept-eula

The installer will then deploy the vcsa and inform you, as soon as everythings ready:

===============================================================================
vCenter Server Appliance installer finished deploying "HOSTNAME_OF_THE_VCSA".
This appliance is a vCenter Server instance with an embedded Platform Services
Controller.
    System Name: IP_OF_THE_VCSA
    Log in as: Administrator@DOMAIN_NAME_OF_VCSA_AD i.e. Administrator@vsphere.local
Finished successfully.
===============================================================================

You can then log into https://IP_OF_THE_VCSA to the vCenter Server (with the SSO Login, i.e. Administrator@vsphere.local) or https://IP_OF_THE_VCSA:5480 to the vCenter Appliance Administration (with VCSA Login, i.e. root and PASSWORD_OF_THE_VCSA_ROOT_ACCOUNT)

Debian 9 Offline Installation Bug

I had to install a Debian 9.3 without having access to the internet, so I downloaded the full size offline install DVD from https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd. As usual, I inserted the first DVD into the DVD Drive, started the install and everything worked fine - except for the moment where it didn't ;): I needed to switch DVDs for some software - and directly afterwards, Debian came to the "Install Grub" step - and asked politly to insert DVD 1 again - which I tried to do - but could not:

For some reason, the Debian GUI Installer had locked the DVD Drive and I could not remove the DVD - could not open it. I then used the Emergency Eject, inserted DVD 1 and closed the Drive again with very mild force.

However, the Installer did not recognize the new DVD and said, that I needed to insert a DVD. Well...

To cope with that problem, just CTRL+ALT+F1 switch to a different Terminal, enter mount /dev/cdrom /cdrom and switch back to the Installer Terminal with CTRL+ALT+F7 (I think it was ;)) - and hit "Continue" - and from that moment on, it worked again and the installation ended successful after the GRUB installation.

 

[VMWare] Get and upgrade ESXi 6.5 "offline" - without paid license

As I wanted to have a very recent version of ESXi, I went to VMWares website and checked out their Products, Free Products, vSphere Hypervisor section. This, however, only presented me with a ESXi 6.5.0a ISO from 02.02.2017 - too old. However, you'll get the much needed free license - so the visit pays off :).

So to get the latest version and updates, you need to go to http://vmware.com/go/evaluate-vsphere-en - and are presented with the 6.5.0 U1 ISO from 27.07.2017 - a lot better. With said image you can then install your server. Even if you had an old 6.5.0a install, you could download the VMware vSphere Hypervisor (ESXi) Offline Bundle - which will upgrade your old 6.5.0 installation to U1 from that site.

After that, you'll need to check out the very useful VMWare ESXi Patch Tracker on https://esxi-patches.v-front.de/ESXi-6.5.0.html. There you can see, which patches are needed to get your ESXi host to the latest version (in my case I only need to apply the 2017-10-05 patch series to get from U1 to latest). So now switch over to https://my.vmware.com/group/vmware/patch#search and look for ESXi 6.5.0 patches. I did find my needed ESXi650-201710001 patch with release date 05.10.2017 - and downloaded it. From the ESXi Patch Tracker I now know, that the Imageprofile of said Update is called ESXi-6.5.0-20171004001-standard and uses the Buildnumber 6765664. I then enabled SSH on the ESXi Host, shutdown all VMs, put the ESXi Host into Maintance mode and uploaded the ESXi650-201710001.zip to a folder on my Datastore datastore01 into a folder I created called ESXiUpdate.

After that, I could execute said update via SSH with the command esxcli software profile update --depot="[datastore01]ESXiUpdate/ESXi650-201710001.zip" --profile ESXi-6.5.0-20171004001-standard

As you can see, it needs to provide the path to the patch file, as well as the Imageprofilename we found out earlier via the ESXi Patch Tracker. After the successful installation, a reboot is need.

As soon as the machine has booted again, login and check if the Buildnumber now matches the Updates Buildnumber. If this is true, disable the Maintenance Mode, restart the VMs and you're good to go.

If other patches need to be applied, you would re-enable SSH  access, not restart the VMs and not disable the Maintenance Mode and just keep on uploading and applying the updates :).

More infos abot the esxcli commands can be found here - and you can still use your free license with ESXi 6.5 which you acquired at the first steps of this weblog - even if you use the most recent patch (luckily!).

And now, get those machines patched ;)!

Rogue One: My first field pentesting

Earlier that year, very - earlier - I had one technician of an international company calling me for some advice. She/He had a problem with the local networking staff and their "modus operandi" regarding network security. The company had a big assembly line and very powerful, automated machinery - which made the leaky security all the more troublesome. My job was to exploit one of those security holes and show - as clearly and easily as possible - said problems - so that they were getting finally fixed.

The first stage of the whole testing was the usual: Reconnaissance. Though, in this case, this was very easily achived, as my contact handed me over parts of the firewall ruleset as well as an access to their office lan. First thing that lit up like a christmas tree: They actually had the production and office networks seperated by a firewall - which is good. For the bad part: They did drop everything. Everything except everykind of ICMP packets. Well. Damn.

Second stage was to create an exploit to that happy little mishap: My contact wanted to be able to bridge office and production networks and access them via the - according to the networking department - water tight secure firewall. The exploit needed to be able to run on a Windows 7 machine as well. With that in mind, I went through different ICMP tunnels: HANS and Dhaval Kapils icmptunnel were the first one to be dropped from that list, as they did not satisfy all constrains. In the end, I choose icmptunnel or short ptunnel. With a bit of manual patching, I could get it to compile and work again on Windows, thanks to the efforts of Mike Miller.

For testing I recreated the network and firewall using a Cisco 1841 and a Cisco 3560 switch. As I needed to integrate ptunnel into the production network, I wanted it to look as innocent and  inconspicuous as possible: So I used a Raspberry Pi 3 and dumped it into a DIN Rail case - then I outfited it with a Power over Ethernet adapter and could serve it network as well as power over said network connection.

The tests worked flawlessly and I even cramped enough speed over ICMP to get some remote desktop working.

 

On to stage three: Attack.

This stage turned out to be way cooler than thought: Due to certain circumstances, we meet at night, 0 dark thirty - you could say - and sneaked through the production line, past workers which did not take much notice in my presence. I inserted the "Rogue Pi" into one closet next to an Siemens Human-Machine Interface and plugged it into the network switch.

Then we left again. Back in the office, I tried to connect to my little helper and was immediately rewarded with a working ICMP tunnel - now transfering an SSH connection as payload. From that moment on, I could connect to a dozend different systems from different vendors in that production network. Last but not least, as "visual" demo, we created a little batch script to start the connection and connect to the Remote Desktop Interface / Human Machine Interface of a very heavy and very unsecured press - now leaving it to our control.

At this point, said connection was only opened in a "read/view only" mode so that - even by accident, we could not harm anyone. We had to bear in mind that this multi-hundred ton press was now at the mercy of our fingertips and we did not wanted to wreck hevoc at all costs - so - if you're conducting field exercises with real "heavy hardware" - find a way to interact safetly with that - before you engage any connection to it.

With this preparation, the technician was able to run the demo in front of the higher ups and finally got the attention, permission and support needed to bring security to a higher standard.

So that effort paid of in the end for the production security of that company - and rewarded me with my first - and hopefully not last - field pentest :).

 

[Dell] Using the Update CDs to get Dell Servers to latest firmware

Dell has an very comfortable way of getting new firmware to nearly all of its server components: The bootable media / ISO or Update CDs. You can find them on this website and very useful. On a basic level, you pick your server, download the ISO, compare the MD5 checksum and burn the ISO onto a DVD. After that, you should get the Servicetag of your server and check for BIOS and iDRAC updates - these should be installed manually first. After that, boot from the DVD and let it install all the needed firmware. Basically, the DVD will cycle through all firmware of components ever installed in the series of your particular server and installs updates if needed. After another reboot, you're done :).

Thanks Dell for being so helpful to your users! 🙂

[Dell] T30 Intel AMT Blank Screen on Ubuntu Fix

The Dell T30 is an awesome little Homeserver, packing a punch with the Xeon E3-1225 V5 - and being affordable at about 399 €. It also comes with Intels Active Management Technology / AMT which is an extension of the horrible Intel Mangement Engine (which was all over the place months ago when some genius figured out how to stop that Man-in-the-Middle-always-on chip with some simple commands) - but quite useful - nonetheless. The good thing about this, is that it acts like an DRAC (Dell) / ILOM (Sun) / IPMI (Supermicro) card - so it is an KVM (Keyboard Video Mouse, not the virtualization thingy this time, sorry ;)) extension which allows you to control the server via network as if you were plugged in directly.

There is an awesome guide from Christian on goNeuland, written in German on howto setup that thing without the need to buy VNC Viewer Plus.

However, my Ubuntu instance came in as blank screen after successfully connecting to the system. In the end, that turned out to be that way, as Ubuntu decided to deactivate the graphics unit - due to no monitor being attached.

Different solutions were talked about herehere and here.

In my case, following helped:

1.) Open your grub, i.e. sudo vi /etc/default/grub file

2.) Add nomodeset to your GRUB_CMDLINE_LINUX_DEFAULT line, so that it would read i.e. GRUB_CMDLINE_LINUX_DEFAULT="reboot=force bootdegraded=true nomodeset" (your commands will vary!)

3.) Save and close the file

4.) Update grub via sudo update-grub

And after a quick reboot, everything worked out :)!