Wednesday, 9 August 2017

"3G MiFi Modem Car Wifi Router", Lithium Batteries and the Internet of Things

The Internet of Things quite simply is when your devices (TV, house alarm, automatic gate, refrigerator, etc) get on the Internet.

The official definition doesn't seem to sound as pithy: "the inter-networking of physical devices, vehicles (also referred to as "connected devices" and "smart devices"), buildings, and other items embedded with electronicssoftwaresensorsactuators, and network connectivity which enable these objects to collect and exchange data".

This post is about the "inter-networking" portion, where your house devices use a gateway to get on the Internet. A "cookie-cutter" gateway can be made very cheaply by using a tiny Linux computer connected via WiFi to your house broadband (ADSL) modem.

I like the Raspberry Pi and the beaglebone. Both are capable of running standard, full-size Linux, especially the Slackware Linux distribution (OK, so I am an old geezer).

These computers are powered from your smartphone charger, and by extension the power bank. Best of all they are cheap. Technically they are overkill as IoT controllers, but they are fun and they are full Linux (you get python!) and hardware gets cheaper every year so unless you cannot afford the electrical power do try them out.

Beaglebone White (left) and Raspberry Pi with PiFace cape
For sustained (and I mean months on end) use, I prefer the beaglebone. The Raspberry Pi despite its many improvements and upgrades use the Broadcom chipsets which have USB ports that are much less reliable. And the Pi uses Broadcom USB for both its copper LAN and its WiFi interface which no amount of clever software can work around.

For some 3 years now I have been running IoT systems at remote sites. This lets me remotely access my systems from my study or my office. It is not always possible to get telephone landline ADSL, but usually there is a cellphone reception that is good enough. I just need to change my gateway to a 3G modem-router.

The first 3G modem routers I used had lithium batteries in them. I did not need the batteries as I had mains power, but they would not start up without the batteries. These batteries tend to fail, often because I had them on all the time in non-airconditioned places.

The D-Link DWR-730 with battery dismounted(top). Notice it is bulging in the middle 
TP-Link TL-MR3040 with attached 3G dongle. Notice the failed battery, which overheated whenever in use.

It was time to get one which did not have a lithium battery and I settled on the no-brand "3G Mifi Modem Car Wifi Router Mini Wireless Routers Unlock Modem with SIM Card Slot". The English seemed a little dodgy but hey, it was cheap (only RM78) and it did not have a lithium battery.


3G Mifi, reallr a WR-706

It arrived without a manual but had a crumpled scrap of paper with 2 lines scribbled on it.



I removed the cover and installed the SIM card with some difficulty: there were 4 positions that fit but only 1 correct position.

3G Mifi with SIM card installed. This is the only correct position
I powered it on from my smartphone charger cable, pressed the power button and ... nothing. But we engineers are bigger than mere manuals, and after much tinkering found that I had to hold down the power button for a few seconds before it will turn on.

From my laptop I looked for a new WiFi hotspot using the command 'iwlist wlan0 scan'

iwlist wlan0 scan
          Cell 03 - Address: 02:03:7F:95:6E:E5
                    Channel:6
                    Frequency:2.437 GHz (Channel 6)
                    Quality=68/70  Signal level=-42 dBm
                    Encryption key:on
                    ESSID:"MIFI_WR706_EE5"
                    Bit Rates:1 Mb/s; 2 Mb/s; 5.5 Mb/s; 6 Mb/s; 9 Mb/s
                              11 Mb/s; 12 Mb/s; 18 Mb/s
                    Bit Rates:24 Mb/s; 36 Mb/s; 48 Mb/s; 54 Mb/s
                    Mode:Master
                    Extra:tsf=0000000001c2176f
                    Extra: Last beacon: 5273ms ago
                    IE: Unknown: 000E4D4946495F57523730365F454535
                    IE: Unknown: 010882848B0C12961824
                    IE: Unknown: 030106
                    IE: Unknown: 0706555320010B1B
                    IE: Unknown: 2A0100
                    IE: Unknown: 32043048606C
                    IE: WPA Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (2) : CCMP TKIP
                        Authentication Suites (1) : PSK
                    IE: IEEE 802.11i/WPA2 Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (2) : CCMP TKIP
                        Authentication Suites (1) : PSK
                    IE: Unknown: DD180050F2020101000003A4000027A4000042435E006232
2F00
                    IE: Unknown: DD640050F204104A0001101044000102103B000103104700
101AFAD653631652ECB2FC1CF677E2E230102100085155414C434F4D4D10230006415236303078102
400012010420001201054000800060050F2040001101100074152364B2D415010080002010E

Aha, MIFI_WR706_EE5 was exactly what was scribbled on that scrap of paper. The next line was 1234567890 which I took to be the password. This lets me produce a wifi configuration file /etc/wpa_supplicant.conf as:

cat /etc/wpa_supplicant.conf
ctrl_interface_group=0
eapol_version=1
ap_scan=1
fast_reauth=1
#country=US
# WPA protected network, supply your own ESSID and WPAPSK here:
network={
  scan_ssid=1
  ssid="MIFI_WR706_EE5"
  proto=WPA RSN
  key_mgmt=WPA-PSK
  pairwise=CCMP TKIP
  group=CCMP TKIP
  psk="1234567890"
  priority=10
}

Next you run the command:
wpa_supplicant -d -Dwext -iwlan0 -c/etc/wpa_daisy.conf -B

You check the results with
wpa_cli -iwlan0 status

And you get:

bssid=02:03:7f:95:6e:e5
freq=0
ssid=MIFI_WR706_EE5
id=0
mode=station
pairwise_cipher=CCMP
group_cipher=TKIP
key_mgmt=WPA2-PSK
wpa_state=COMPLETED
ip_address=192.168.1.32
address=c8:ff:28:27:7d:2b
uuid=29de4b6d-135e-5be0-a3c3-0db44f618230

Next you do:

dhclient -v -1 wlan0
Listening on LPF/wlan0/c8:ff:28:27:7d:2b
Sending on   LPF/wlan0/c8:ff:28:27:7d:2b
Sending on   Socket/fallback
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 6
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 14
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 18
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 16
DHCPREQUEST on wlan0 to 255.255.255.255 port 67
DHCPOFFER from 192.168.1.1
DHCPACK from 192.168.1.1
bound to 192.168.1.32 -- renewal in 33585 seconds.

And we have liftoff! The last output showed us that the 3G Mifi's IP address was 192.168.1.1, so I fired up my firefox browser and typed into it:
http://192.168.1.1

And got a rude shock- it's setup screen was in Chinese! Engineer or not, it is time to RTFM

Screenshot of the language change screen

That explains the dodgy English. If you moved the mouse around the browser screen it sometimes tells you(at the bottom of the browser) what is clickable. For example right at the top, the white-and-green icon resulted in  'http://192.168.1.1/Userguide_zh_v2.pdf'. Clicking on that produced a manual, but in in Chinese. 

Now websites (specifically HTML) often use languages codes to tell the browser what language to use, and 'zh' means Chinese, so I typed into the browser 'http://192.168.1.1/Userguide_en_v2.pdf' and sure enough an English manual popped up. You can download it here.

From there I discovered that the admin password was 'password', and this unlocked the other screens for me. Now with the manual, I calmed down enough and remembered I am Chinese too: the bit you want to click on is the third horizontal white bar from the top. That is a drop-down menu with 3 selections. Select the topmost one for English. Next press the left button at the bottom of the screen. It should now look like this:


And this is what the main screen should have looked like:


And there you have it. Set the hotspot name and password you want and you have a WiFi gateway for your beaglebone IoT, which is the subject of a later post, so do check in here again.

Happy trails.

Sunday, 23 July 2017

Klang Valley MRT Opens, a Project Ends


Page 3 of the Sin Chew

17 July 2017 was opening day, and it was supposed to be a doddle. We had finished a month ahead of schedule; but then they ordered 4 more escalators at the last minute.

The fastest way to get around was to use the MRT itself. On opening day I finished up at Station 5 and was expecting to take the train to Station 26 but there was a snag: the trains did not run all the way from Stations 5 to 34. The new segment only went from 16 (Museum Negara) to 34 (Kajang).

Station 5

So at opening time 16:00 hours I was at  Museum Negara and went through as fast as I could. A reporter took a picture as I went through and the next day, it made page 3 of the local Chinese paper, the Sin Chew Jit Poh

Post Project Blues

Project C1313K started in 2013 and required remote control of over 200 escalators over 34 stations spanning an area of over 50km. The feature did not exist so it would have to be designed, built and tested on the fly. 
Senior R&D Engineer Edward Boo at ease

For the last 2 years we ate, slept and dreamed C1313K. There were setbacks, attrition, additional requirements, variation orders, but these are teething pains. You focus: eyes on the prize...

The team in 2015


You get used to being occupied by a design for years. Then it is over. OK, it succeeded, it works and is delivered on time and we get to fight another day. There is the high, the celebration, the adrenaline rush.  

Northern Zone passed Field Tests Oct 2016
Southern Zone passed field tests May 2017


So where do you go from there? It is strange not to think about C1313K when you get up or just before you go to sleep. Welcome to the Post Project Blues.

 

Reprieve

 And then you get days like these, 18 July 2017 when you make page 3 of a national paper. Did I say life was good?

Moving On

Time for MRT2 (KVMRT SSP Line) then. Goodbye, C1313K.

But with the word the time will bring on summer,
When briers shall have leaves as well as thorns,
And be as sweet as sharp. We must away;
Our wagon is prepared, and time revives us:
All's well that ends well; still the fine's the crown;
Whate'er the course, the end is the renown.
- William Shakespeare


Sunday, 16 July 2017

Webserver Follies

In the winter of 1996 my friend and former boss Wiljan Derks introduced me to Linux. By the following spring I picked up a Walnut Creek Slackware 1.0 CDROM and installed it from a stack of some 13 3.5" floppy disks. My desktop was a 33MHz Zeos 80486DX souped up to a giddy 66MHz with an 80486DX2 CPU



I had a website, running at tripod, back then one of the first sites to offer free webhosting. Next was a Malaysian pay webhosting site costing some RM300 a year, which proved short-lived. After the site raised its charges by a few hundred percent, and with 3 years of Linux under my belt I hosted the website at home on my trusty Zeos over a 512Kbps broadband line. It was a joy- Linux was a natural fit as a webserver, and without needing to share the server with other websites, performance jumped.

From 1998, home was on top of a little hill in Seremban, which seemed to be a magnet for lightning.No way a little bit of natural static electricity was going to stop me, right? For 20 years I fought the good fight. First I just repaired the procession of stricken modems, routers, network hubs, desktops and  implacably put them back on line. Then I realized that the lightning caused so many power outages I needed an uninterruptible power supply.

The UPS solved the power outage problem, but made the lightning problem worse. With the web server running through the storm the lightning strokes frequently ran through the telephone landline into my power grid Earth, often passing through the surge arrestors in the modem. This made for a lot of dead modems. Sometimes the strikes hit the lamp or telephone post in my compound(don't even talk to me about the gatepost lights) and even damaged monitors, network switches and network cards.

WiFi brought some hope - there is now no direct link between the server and the modem. True, modems still died at the usual rate of some 4 or 5 a year but the server was usually fine. This meant the UPS had to be up-sized especially if the storm occurred on a weekend and we were not at home to reset the mains breaker. Even with an out-sized UPS, the battery replacement costs were roughtly RM800 over 2 years. It made more sense to downsize the server to the ARM-based beaglebone. The Raspberry Pi seemed even better, but try as I might it would not run WiFi reliably.      

 Currently the web server runs on a Beaglebone original, a 720MHz 256MB ARM Cortex-M3 (Zeos was 66MHz 16MB and used bucketloads of power), hacked to run Slackware. A WiFi modem supplies 4MBps bandwidth (512Kbps uplink) and the hacked APC Matrix 3000 can pretty much power the setup for a week, but Fate intervened.




After so many years, the wife have had enough of the lightning strikes taking out the modems and interrupting her Facebook sessions. She had taken to disconnecting the telephone line from the modem before the storms which saves the modems but do not do much for server uptime.

So it is time for paid webhosts again. I signed up for Amazon's AWS and after a full year's free trial it is time to move the website, this time to AWS with docker and Wordpress, the latter to to sex up the 20-year old website a little.

About time, don't you think?

Saturday, 1 July 2017

The "TWO Groups" Section Switch and the Internet of Things

The wife bought a fancy lamp for the gatepost which changes color when the power switch is flipped off and on quickly. I got a chance to look at it when it got damaged by a nearby lightning bolt. It turned out there were two LED lamps inside, one white and the other yellow. An electronic module switches one or the other or both based on the number of times you flipped the power off and on.


Now this is a great way of powering an Internet of Things remote device on the cheap. Say you want to install a WiFi camera and you want to be able to switch it on together with a nearby lamp (for night use), or to turn on just the camera without the lamp, or just turn the light on without turning on the camera. You install this Section Switch so that it powers both the lamp and the camera.

I dismantled the gatepost lamp, and the wife took it back to the shop where she bought it, where she had it repaired for RM20 (less than USD5). The shop replaced the section switch, and returned the defective section switch as proof of repair. You can buy it online (or at your friendly lighting shop) for RM18.

Warning - do not attempt this yourself unless you have had specialist training in switch-mode power supplies. There are lethal voltages in this module.

I thought I would fix the Section Switch just to see how it worked. It is easily opened- just press firmly at the base of the long side of the module and the base plate pops out.



The lights are switched by two 12V relays controlled by an integrated circuit, the JY2608. I am afraid the datasheet is in Mandarin, but don't let that stop you. The original website is a bit spotty- if you cannot get the datasheet place a comment here and I'll email you my copy.

The datasheet has a sample circuit for the JY2608 and it looked like the manufacturer has copied it lock stock and barrel. Here is the sample circuit from the datasheet:



Now the problem with the Section Switch was when switched, the lights took a very long time (sometimes hours) to come on. So we know the problem is related to the power section.

I powered up the defective module with this setup:


Note the use of an isolation transformer (the box with the white power dial) and and a portable ELCB for safety. The Section Switch has no transformer, so the semiconductors are connected directly to the mains power - in this case 230Vac. Some types of faults can cause the module to explode or catch fire if connected directly. Here, the isolation transformer limits the mains power to less than 20W. An isolation transformer can break the Earth connection to your house ELCB, so use an extra ELCB to protect yourself.

I have also secured the wires to a strip of screw terminals for extra safety.

OK, back to the input power problem. I measured the input power to the IC (the JY2608 is a 12V device) at D1 the 12V zener diode's anode and it read 5.8V. That is very far from the 12V it needs. Actually the JY2608 will work at 2V but the relays need at least 9V to switch.

I replaced the 12V zener with an 1N4742A and powered on, but the voltage stayed at 5.8V. So it is not the zener diode but something closer to the input power.

A closer look at the circuit showed that the bridge rectifier (four diodes in the shape of a diamond) is powered by C1 and R1 in parallel, a 1.5uF capacitor and 330KOhm resistor. I tested the resistor using the multimeter (the black and orange thing in the foreground) and it reads correctly at 330K.

Next I replaced the capacitor with a 1uF capacitor I had handy, and now the Section Switch powers up nicely at 12V and the relays clicked when I flipped the mains power off then on. 230V appeared on both outputs so the repair is done.

Now the repair costs some RM3 for the 1uF 400V film capacitor, and it only cost RM18 to buy, not to mention the hazards involved. But that is a little bit of electronics saved from the scrap heap.

The next step would be to hook it up to power an IoT device, so stay tuned!

Monday, 26 June 2017

IOMMU PCI Express USB3 Controller Passthrough


One of the best things about Virtual Machines for me is I can bake in support for legacy systems like Windows XP. I needed to support the Renesas SH2 microcontroller and needed the Windows XP version of their PG-FP5 programmer. I currently use a Dell Inspiron 1420, and it is getting a little frail - the battery is failing and it is a security risk to any LAN. I airgapped it, and the constant copying of program SH2 files via thumbdrive is getting tiresome.

One way to link up with the PG-FP5 was an old-fashioned windows COM which do not exist anymore in the new laptops. The PG-FP5 version I have does not work with USB serial port dongles, but they have a USB port. The idea is to passthrough an entire USB controller to my Windows XP qemu Virtual Machine.

I picked up some generic USB3 PCI Express controller(Top: Via VL805. Bottom: NEC uPD720200):


With the NEC uPD720200 adapter installed, just do the usual (see previous blog post):

root@crosshair:/home/heong$lspci | grep USB
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB EHCI Controller
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB OHCI2 Controller
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB EHCI Controller
04:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)
05:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)

There are 2 identical NEC controllers here.

root@crosshair:/home/heong$lspci -nn | grep 05:00
05:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller
 [1033:0194] (rev 03)
root@crosshair:/home/heong$lspci -nn | grep 04:00
04:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller
 [1033:0194] (rev 03)

Indeed, the Crosshair IV Formula has a builtin USB3 controller. This complicates things a little. We need to passthrough the correct controller. I happened to have a Sandisk thumbdrive handy and with it plugged into the new PCI card, we do:

dmesg -T | tail

And we get something like:

[Mon Jun 26 15:28:08 2017] usb 10-2: new high-speed USB device number 2 using xh
ci_hcd
[Mon Jun 26 15:28:08 2017] usb 10-2: New USB device found, idVendor=0781, idProd
uct=5572
[Mon Jun 26 15:28:08 2017] usb 10-2: New USB device strings: Mfr=1, Product=2, S
erialNumber=3
[Mon Jun 26 15:28:08 2017] usb 10-2: Product: Cruzer Switch
[Mon Jun 26 15:28:08 2017] usb 10-2: Manufacturer: SanDisk
[Mon Jun 26 15:28:08 2017] usb 10-2: SerialNumber: 4C532000020924119064
[Mon Jun 26 15:28:08 2017] usb-storage 10-2:1.0: USB Mass Storage device detected
[Mon Jun 26 15:28:08 2017] scsi host11: usb-storage 10-2:1.0
[Mon Jun 26 15:28:09 2017] scsi 11:0:0:0: Direct-Access     SanDisk  Cruzer Switch    1.20 PQ: 0 ANSI: 5
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] 15633408 512-byte logical blocks:
(8.00 GB/7.45 GiB)
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Write Protect is off
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Mode Sense: 43 00 00 00
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Write cache: disabled, read cache:
 enabled, doesn't support DPO or FUA
[Mon Jun 26 15:28:09 2017]  sdg: sdg1
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Attached SCSI removable disk

We now know that it is in USB device 10-2. But which PCI device it it in?

root@crosshair:/home/heong$lsusb -v 2>/dev/null | grep '^Bus\|iSerial'
Bus 003 Device 002: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer
  iSerial                 3 058F312D81B
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:00:16.2
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  iSerial                 1 0000:00:16.0
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  iSerial                 1 0000:00:14.5
Bus 002 Device 002: ID 0781:5572 SanDisk Corp.
  iSerial                 3 4C532000020924119064
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:00:13.2
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  iSerial                 1 0000:00:13.0
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:00:12.2
Bus 004 Device 002: ID 046d:c247 Logitech, Inc.
  iSerial                 0
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  iSerial                 1 0000:00:12.0
Bus 011 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
  iSerial                 1 0000:04:00.0
Bus 010 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:04:00.0
Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
  iSerial                 1 0000:05:00.0
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:05:00.0


That would be Bus 010 0000:04:00.0. Now we do the usual

root@crosshair:/home/heong$lspci -nn | grep 04:00
04:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller
 [1033:0194] (rev 03)

root@crosshair:/home/heong$modprobe vfio
root@crosshair:/home/heong$modprobe -v vfio_iommu_type1
root@crosshair:/home/heong$modprobe -v vfio_pci
insmod /lib/modules/4.11.2/kernel/drivers/vfio/vfio_virqfd.ko
insmod /lib/modules/4.11.2/kernel/drivers/vfio/pci/vfio-pci.ko
root@crosshair:/home/heong$echo '0000:04:00.0' >  /sys/bus/pci/devices/0000:04:0
0.0/driver/unbind
root@crosshair:/home/heong$echo '0x1033 0x0194' > /sys/bus/pci/drivers/vfio-pci/
new_id

The half-size DVD-ROM was a problem for my vertically mounted drive, so I ripped it in my laptop's drive thus:
root@aspireF15:/home/heong/ECI$dd if=/dev/dvd of=usb3pcie_nec.iso

And copied the resulting ISO file into the VM host, the Crosshair Formula IV.

Next you make a fake DVD for the qemu VM:
root@crosshair:/home/heong/ECI$losetup /dev/loop0 ./usb3pcie_nec.iso

Your qemu command should be something like:
qemu-system-x86_64 -cpu host,kvm=off -enable-kvm -m 4G -balloon none -bios ./seabios
/out/bios.bin -hda XP-12GB.img -rtc clock=host,base=localtime -device vfio-pci,host=04:00.0 -cdrom /dev/loop0

You install the Windows XP USB3 driver from the DVD-ROM and it should work with no trouble. Indeed I installed the USB drivers for the Renesas PG-FP5 and that worked first time too.



That is the sanitized version, of course. In the real world, rather than buying the NEC PCI Express card, I tried passing through the onboard NEC USB3 controller, 05:00.0. The Windows XP driver installed correctly and the VM image rebooted successfully, but it would not recognize devices plugged into it.

Next I ran to the nearest computer store and bought the one PCI Express USB3 card they have on hand, the Via VL805. This went a lot worse- the Windows XP driver locked up the VM during installation, and the VM locked up on all subsequent reboots.

With some care the next PCI Express USB3 controller I chose had a different chipset, the NEC  uPD720200 with the happy results detailed earlier.

So if at first you don't succeed, try again ... Happy Trails.


Saturday, 10 June 2017

Internet of Things, Legacy Industrial Applications, and PCI Passthrough

It's been a while since the last blog- been busy with a huge order of over 200 escalators for the above-ground stations of the Klang Valley Mass Rapid Transit (KV-MRT).


Building Management System


Elevator & Escalator Monitoring & Control System in Visual Basic

BMS, EMS, Modbus (and other acronyms)


The new twist to an otherwise standard escalator is the requirement for Modbus input and output. The intention is to connect  them to a Building Management System via BACNet. Because all 31 stations are linked, an operator can start and stop the escalators remotely, generally not a good idea from a safety point of view. Still, those are the customer's requirements, and with a proper standard operating procedure this allows for very quick mass evacuations.

Our existing Escalator Management System runs on the 16-year old Windows XP and worse, on Visual Basic. BMS support was via a proprietary MS Elevator protocol. At design stage back in 2011, given the delivery deadline, and the fact that all new designs and development systems run on Linux, I added a Modbus-MS protocol translator Windows program, written in python. This linked to the BMS via rs-485, as it was risky to connect Windows XP to a WAN.

It is now clear that this is still a security risk as nearly everyone now have a smartphone with a mobile dataline and it is straightforward to connect the Windows XP system to WAN with a USB cable. Indeed the rail operator staff have begun to do that, mainly to recharge their phones.

Both Windows XP and Visual Basic have been orphaned by their creator, Microsoft. To harden it, we can install an up-to-date Linux distribution and run Windows XP and the EMS application in a virtual machine. This hardens it to the extent a CAT6 Ethernet Modbus link is possible. The disadvantage is IO throughput now drops some 30%, as the virtual machine is an additional layer of code to run through.

Native Virtualization, IOMMU and PCI Passthrough


Back in 2011 AMD published a specification for IOMMU which a virtual machine like Qemu-KVM can exploit to improve IO throughput, together with hardware-assisted(native) virtualization. We use PCI-SIG Single Root I/O Virtualization, or PCI passthrough. It did not work out for us back then, but from 2016 or so PC Gamers have been reporting success with VGA passthrough so it is time to revisit it.


The Rig

Native Virtualization, and IOMMU in particular require a very specific subset of hardware. I used my old 2011 rig, an Asus Crosshair Formula IV, with an AMD Phenom II X4 945 and 16GB RAM. The VGA cards are an old (2011) Radeon HD 5550 and a new Nvidia Gigabyte Geforce GT710. The rs485 link was through a Moxa CP114EL. Only the PCI Express slots were used. The Nvidia card was used by the Linux host while the old Radeon card was passed through to the Windows XP guest. This is because there were no XP drivers for the newer card.


The Code

I used Linux Slackware 14.2, with the kernel upgraded to 4.11.2. Qemu-KVM version is 2.9.0. You will need a pre-installed Qemu image of Windows. You will also need seabios. I used David Yates' excellent guide. If you do not have a Windows image, David Yates' guide has the procedure to generate one.

root@crosshair:/home/heong/ECI$lspci | grep VGA
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570]
08:00.0 VGA compatible controller: NVIDIA Corporation Device 128b (rev a1)

root@crosshair:/home/heong/ECI$lspci -nn | grep 03:00
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] [1002:68d9]
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] [1002:aa60]

root@crosshair:/home/heong/ECI$lspci  | grep Moxa
02:00.0 Serial controller: Moxa Technologies Co Ltd CP-114EL (4-port RS-232/422/485 Smart PCI Express Serial Board)

root@crosshair:/home/heong/ECI$lspci  -nn | grep 02:00
02:00.0 Serial controller [0700]: Moxa Technologies Co Ltd CP-114EL (4-port RS-232/422/485 Smart PCI Express Serial Board) [1393:1144]


Unlike David, I use the lilo bootloader, and its configuration file /etc/lilo.conf needs to be changed to:
append=" vt.default_utf8=0 quiet splash modprobe.blacklist=radeon"

Update the boot track using liloconfig and re-use the existing /etc/lilo.conf.

For clarity, rather than use a bash script, I have the following set of manual commands:

modprobe -v vfio
modprobe -v vfio_iommu_type1
modprobe -v vfio_pci

And you follow that with:
echo '0000:03:00.0' >  /sys/bus/pci/devices/0000:03:00.0/driver/unbind
echo '0x1002 0x68d9' > /sys/bus/pci/drivers/vfio-pci/new_id
echo '0000:03:00.1' >  /sys/bus/pci/devices/0000:03:00.1/driver/unbind
echo '0x1002 0aa60' > /sys/bus/pci/drivers/vfio-pci/new_id
echo '0000:02:00.0' >  /sys/bus/pci/devices/0000:02:00.0/driver/unbind
echo '0x1393 0x1144' > /sys/bus/pci/drivers/vfio-pci/new_id

Note the numbers 1002:68d9,1002:aa60, and 1393:1144 are from the lspci commands earlier.



The Windows XP device driver for the Radeon card is on a DVD. The Moxa CP-114REL driver was on a different DVD, which needed to be swapped in and the following command repeated:

My Qemu-KVM command is:

qemu-system-x86_64 -cpu host,kvm=off -enable-kvm -m 4G -balloon none -bios /home
/heong/ECI/bios/seabios/out/bios.bin -hda XP-12GB.img -hdb scratch--4GB.img -cdrom /dev/dvd -device vfio-pci,host=03:00.0,multifunction=on -device vfio-pci,host=03:00.1 -rtc clock=host,base=localtime -device vfio-pci,host=02:00.0

And it worked first time, no fuss. The first thing to do is to update the wiki and check off the Asus Crosshair Formula IV :7).

PCI (VGA & Moxa Multiport) Passthrough

The IO throughput is very near bare metal and has been running 16 hours.

Internet of Things

The Linux Slackware host is also a LAMP Stack capable of supporting smartphone applications as a server. This would make the escalators part of the Internet of Things (IoT). Even the original Windows XP Modbus translator in python can be run with minimal modification from the Linux side.

Security


The Linux VM can be locked down for only https and ssh access. In particular, the virtual machine can be configured with a virtual network with to give Windows XP only ssh access and only from the Linux host. But that is the another blog post. Stay tuned!


Saturday, 12 November 2016

Dirty Cow and Slackware


Dirty cow, or CVE-2016-5195 is a formidable exploit. It works in most Linux versions including Android. Here is a news article for easier reading.

I downloaded the exploit here and followed the instructions in dirtyc0w.c:

####################### dirtyc0w.c #######################
$ sudo -s
# echo this is not a test > foo
# chmod 0404 foo
$ ls -lah foo
-r-----r-- 1 root root 19 Oct 20 15:23 foo
$ cat foo
this is not a test
$ gcc -pthread dirtyc0w.c -o dirtyc0w
$ ./dirtyc0w foo m00000000000000000
mmap 56123000
madvise 0
procselfmem 1800000000
$ cat foo
m00000000000000000
####################### dirtyc0w.c #######################

The program compile was no trouble:

~/dirtycow/dirtycow.github.io-master$gcc -pthread dirtyc0w.c -o dirtyc0w

Setting up for the exploit. Use your root account:

/home/heong/dirtycow/dirtycow.github.io-master$echo 'This is not a test' > foo
/home/heong/dirtycow/dirtycow.github.io-master$chmod 0404 foo
/home/heong/dirtycow/dirtycow.github.io-master$ls -lah foo
-r-----r-- 1 root root 19 Nov 13 12:46 foo
/home/heong/dirtycow/dirtycow.github.io-master$cat foo
This is not a test
/home/heong/dirtycow/dirtycow.github.io-master$exit
exit
~/dirtycow/dirtycow.github.io-master$cat foo
This is not a test

And the attack worked, when run as non-root user:

~/dirtycow/dirtycow.github.io-master$./dirtyc0w foo youdirtycow
mmap 7fd267f8f000

madvise 0

procselfmem 1100000000

~/dirtycow/dirtycow.github.io-master$cat foo
youdirtycow a test

Even worse, it worked on my current Linux kernel in use, 4.6.2. This called for a new kernel. The newest stable kernel was 4.8.7 

$rm /usr/src/linux
$ln -s /home/heong/Linux/linux-4.8.7 /usr/src/linux
$cd /usr/src/linux
/usr/src/linux$cp -ap /home/heong/Linux/linux-4.6.2/.config .
/usr/src/linux$make oldconfig
/usr/src/linux$make -j20 bzImage
/usr/src/linux$make -j20 modules
/usr/src/linux$make modules_install

Now copy the new kernel to the boot directory to be loaded:
/usr/src/linux$cp .config /boot/config
/usr/src/linux$cp System.map /boot/
/usr/src/linux$cp -a arch/x86/boot/bzImage /boot/
/usr/src/linux$liloconfig

Reboot the laptop:
/usr/src/linux$reboot

Again setting up for test as root:

/home/heong/dirtycow/dirtycow.github.io-master$echo 'This is not a test' > foo
/home/heong/dirtycow/dirtycow.github.io-master$chmod 0404 foo
/home/heong/dirtycow/dirtycow.github.io-master$exit
exit
~/dirtycow/dirtycow.github.io-master$cat foo
This is not a test
~/dirtycow/dirtycow.github.io-master$ls -lah foo
-r-----r-- 1 heong users 19 Nov 13 15:21 foo

Now Dirty Cow is rebuffed:

~/dirtycow/dirtycow.github.io-master$./dirtyc0w foo thishadbetterwork
mmap 7f695f9c7000

madvise 0

procselfmem 1700000000

~/dirtycow/dirtycow.github.io-master$cat foo
This is not a test

Now I only have to upgrade every Linux computer I own ... :7(