Thursday, 28 December 2017

IoT Bluetooth Autogate using Modbus


The IoT Bluetooth Relay Board tests well enough with a simple python script, but every time the script finishes bluetoothctl drops its connection to the HC-06. When you run the test script again, it reconnects to the HC-06 well enough, but the connection takes a variable number of seconds, and sometimes the script fails to connect. For an application like the autogate this is not acceptable.

However if the script does not exit the connection is reliable over many days, so the script should not exit. That means a server (Linux daemon) process, and its accompanying client script. There is one such python solution and that is pymodbus, a python implementation of Modbus client and server.

Modbus for an autogate? Modbus is a popular industrial communications protocol. Isn't that overkill?. It probably is.

Or is it? As we have seen the remote sensor/actuator portion is done using bluetooth device and a Microchip PIC18F14K50, which consumes 10mA at 5V. Modbus lives on the IoT gateway, ie the bluetooth master which happens to be an old laptop running Slackware Linux. Pymodbus makes my homebrew autogate remote opener compatible with IoT for Industry. The client python script will then be run by a PHP script from the laptop's apache webserver, in the same way as the Raspberry Pi Robot platform.

Security is via the WPA WiFi password on my home WiFi router. Notice that this system will still work without a broadband connection, as long as your smartphone is within range of the WiFi. The HC-06 will not accept a second bluetooth connection once it is paired to the laptop, so access is via the WiFi alone. With a ADSL connection this is a true IoT, able to accept commands from the Internet.

Remote autogate operation may make sense if you say, want to let the gardener into the yard, or the electricity/water utility person to read the meter. In my case my dogs will probably terrorize the meter reader   before galloping off to poop in the neighbour's yard. So its main advantage is an extended-range autogate remote.

The installation for Slackware is probably subject for another post, but say you had it installed. There is a sample program pymodbus-master/examples/common/updating-server.py which we can use as a template.

The first few lines of code are important:
#---------------------------------------------------------------------------#
# import the modbus libraries we need
#---------------------------------------------------------------------------#
from pymodbus.server.async import StartTcpServer
from pymodbus.device import ModbusDeviceIdentification
from pymodbus.datastore import ModbusSequentialDataBlock
from pymodbus.datastore import ModbusSlaveContext, ModbusServerContext
from pymodbus.transaction import ModbusRtuFramer, ModbusAsciiFramer

#---------------------------------------------------------------------------#
# import the twisted libraries we need
#---------------------------------------------------------------------------#
from twisted.internet.task import LoopingCall

Next we import our bluetooth module, which shall look suspiciously like our previous python test script.

#---------------------------------------------------------------------------#
# communicate with bluetooth rs-485 pic18f14k50 2017-12-19
#---------------------------------------------------------------------------#
import autogate_bluetooth

The main server loop code is simply; we will add our bluetooth code later:
#---------------------------------------------------------------------------#
# define your callback process
#---------------------------------------------------------------------------#
def updating_writer(a):
    ''' A worker process that runs every so often and
    updates live values of the context. It should be noted
    that there is a race condition for the update.

    :param arguments: The input arguments to the call
    '''
    log.debug("updating the context")
    context  = a[0]
    register = 3
    slave_id = 0x00
    address  = 0x10
    values   = context[slave_id].getValues(register, address, count=5)
    values   = [v + 1 for v in values]
    log.debug("new values: " + str(values))
    context[slave_id].setValues(register, address, values)

Next there is our Modbus database, which we reduce from 100 to 16 to reduce memory usage, a initialize to 0 (except for the 2 bits reversed logic for the Arduino relays):
#---------------------------------------------------------------------------#
# initialize your data store
#---------------------------------------------------------------------------#
store = ModbusSlaveContext(
    di = ModbusSequentialDataBlock(0, [0]*16),
    co = ModbusSequentialDataBlock(0, [True, True, False, False]*4), # True because relay logic reversed
    hr = ModbusSequentialDataBlock(0, [i for i in range(16)]),
    ir = ModbusSequentialDataBlock(0, [45]*16))
context = ModbusServerContext(slaves=store, single=True)


Jumping to the end of the file for now, we have:
if __name__ == "__main__":
  #---------------------------------------------------------------------------#
  # run the server you want
  #---------------------------------------------------------------------------#
  looptime = 0.2 # 
  loop = LoopingCall(f=updating_writer, a=(context,))
  loop.start(looptime, now=True) # initially delay by time

  StartTcpServer(context, identity=identity, address=("localhost", 5020)) #2017-12-19

Note the loop time has been speeded up from 5s to 0.2s. StartTcpServer() is set to use the non-root network port 5020, which our client script will need to listen to.

That is all there is to it. Most of the pymodbus server code is done. Now for that bluetooth code in ./autogate_bluetooth.py, which I have put in the same directory so that it imports correctly. Here it is in full:

#!/usr/bin/python
import serial
from time import localtime, strftime, sleep

'''
Sends bluetooth rs-485 PIC18F14K50 commands and receives the replies.    
Usage:

import autogate_bluetooth.py
  ./autogate_bluetooth.py             to test
  Uses pyserial module
  The ems command is single-char, reply is a double sequence of 3 binary bytes
  followed by check strings
  Replies to 'z' are 2 3-byte analog input readings. 
'''

port1 = serial.Serial('/dev/rfcomm0',timeout= 1) #2017-12-19 HC-06 bluetooth   
print 'Waiting for bluetooth connect ...',
sleep(8) # Wait for bluetooth to connect
print 'Done'

cmd = ['z', '1', '2', '3', '0' ]
ChkStr1 = 'HCM'
ChkStr2 = 'LJP'

import struct
import binascii

def check_frame(reply):
  if reply[2:5] != 'HCM':
    print 'Checkbyte HCM Fail', reply
    return 0
  if reply[7:10] != 'LJP':
    print 'Checkbyte LJP Fail', reply
    return 0

  return 1
def ParseReply(reply):
  Battery_Voltage=int(format(ord(reply[0]), '02x'), 16)*256 + \
                  int(format(ord(reply[1]), '02x'), 16)
  Calibrated_Battery_Voltage = Battery_Voltage * 13.75 / 689
  PIC_Voltage=int(format(ord(reply[5]), '02x'), 16)*256 + \
              int(format(ord(reply[6]), '02x'), 16)
  Calibrated_PIC_Voltage = PIC_Voltage * 13.75 / 689

  return [int(format(ord(reply[0]))), int(format(ord(reply[1]))), \
          int(format(ord(reply[5]))), int(format(ord(reply[6]))) ]
  #return [Calibrated_Battery_Voltage, Calibrated_PIC_Voltage] 

def Send_Cmd(cmd):
  # sleep(1)
  port1.write(cmd)
  port1.flush()
  reply = port1.read(80)
  if reply != "":
    print 'Reply:', reply, 'len', len(reply)
  # for j in range(0, len(reply)):
  #   print format(ord(reply[j]), '02x'),
  # print '...'

    if (check_frame(reply) != 0):
      data = ParseReply(reply)
    else:
      data = 0
  else:
    print '... Noreply to cmd', cmd
    data = 0
  return data

import sys

# main program loop
if __name__ == "__main__":

  if (len(sys.argv) == 2):
    Send_Cmd(cmd= sys.argv[1])
  else:
    for i in cmd:
      answer = Send_Cmd(i)
      print 'result', answer

  port1.close()

The main function here is Send_Cmd(). The PIC18F14K50 has been programmed to accept the commands 'z' (read voltages), '0' (both relays on), '1' (first relay on), '2' (second relay on) and '3' (both relays off).

Now let us go back to updating_server.py so we can modify the main server loop to communicate with the HC-06. We need to add a new module update_output() to read the Modbus database. If there is a change (caused by the client script requesting the Arduino relay to turn on) it will then send the correct command to the HC-06. The code is:

import struct

def get_muxoutputBuf():
    register = 1      # digital output (modbus 'coils')
    slave_id = 0x00
    address  = 0
    values   = context[slave_id].getValues(register, address, count=8)#from 48
    # print 'values=', values
    return values


def get_muxscratchBuf():
    register = 1      # use holding registers for convenience
    slave_id = 0x00
    address  = 0x0008
    values   = context[slave_id].getValues(register, address, count=8)
    return values

def set_muxscratchBuf(values):
    register = 1      # use holding registers for convenience
    slave_id = 0x00
    address  = 0x0008 # used as scratch buffer
    context[slave_id].setValues(register, address, values)
    return values

def update_output(a): # new function 2016-08-10 09:15
    #context = a
    str = 'z'
    log.debug("Checking the output buffer")
    scratchBuf = get_muxscratchBuf()
    #print 'scratchBuf', scratchBuf
    outputBuf = get_muxoutputBuf()
    #print 'outputbuffer', outputBuf

    data = [0]*1
    #print 'data length', len(data),

    # pack the bits into the integer/byte array 2015-08-23
    for i in range(0, len(data)*8):
        j = i / 8
        k = i % 8
        # print 'j', j
        if outputBuf[i] == True:
            data[j] |= 1 << k
        else:
            data[j] &= ~(1 << k)

    datastr= [format(data[i], '02X') for i in range(0, len(data))]
    str = datastr[0][1] # 2017-12-20

    #print 'data is',data,'datastr',datastr,'output Cmd data string', str
    if (scratchBuf != outputBuf):
        autogate_bluetooth.Send_Cmd(str) # 2017-12-21 note answer is not stored
        print 'Hey, it is here!!!!'
        print '!!!!!!!'
        print '.......'
        print 'setting scratchBuf to', outputBuf
        set_muxscratchBuf(outputBuf)
        print 'scratchBuf', scratchBuf
    '''
    else:
        print 'scratchBuf', scratchBuf,'outputBuf', outputBuf
    '''

    return str

Next there is the code to update the Modbus database with the raw values read from the HC-06. These are the 2-byte values from the analog inputs. There are two inputs that makes 4 buyes in total.

def update_database(context, answer=[]):
    ''' Updated 2017-12-21
    raw = []
    for byte in answer:
      raw += byte
    print 'raw =', raw
    '''

    register = 3 # input register. *Not* 4
    address = 0; # 2017-12-20
    values   = context[0x01].getValues(register, address, count=4)# 2017-12-20
    print 'stored values=', str(values)

    log.debug(str(address) + " input register values: " + str(answer))
    print 'address=', address, ' answer: ', answer

    #context[0x01].setValues(register, address, raw) # 0x01 replaced slave_id
    context[0x01].setValues(register, address, answer) # 0x01 replaced slave_id

    return

battery_scan_interval = 3600 # set at 1 hour

The new server loop now becomes:

def updating_writer(a):
    global battery_scan_interval

    # read battery voltage every hour
    if battery_scan_interval == 3600:
        answer = [0*2] # 2017-12-19
        answer = autogate_bluetooth.Send_Cmd('z') # 2017-12-20
        if answer != 0:
            print '\n\n Cmd z, voltages are', answer
            update_database(context, answer)
            CPU_Voltage = answer[0] * 256 + answer[1]
            Calibrated_CPU_Voltage = (CPU_Voltage * 5.0) / 1024
            Battery_Voltage = answer[2] * 256 + answer[3]
            Calibrated_Battery_Voltage = (Battery_Voltage * 13.53) / 652
            print 'CPU', answer[0] * 256 + answer[1], 'Battery', \
                  answer[2] * 256 + answer[3]
            print 'CPU', "{:4.2f}".format(Calibrated_CPU_Voltage), \
                  'Battery', "{:4.2f}".format(Calibrated_Battery_Voltage),\
                  'Volts',strftime("%Y-%m-%d %H:%M:%S", localtime())

        else:
            print 'no answer'
        battery_scan_interval = 0
    else:
        battery_scan_interval += 1

    str = update_output(a) # 2017-12-20


And that is it. You run the server from a command line (remembering to run bluetoothctl and 'rfcomm bind /dev/rfcomm0' first) :

$python ./autogate_server.py
Waiting for bluetooth connect ... Done
Reply: ûHCM¥LJP len 10

 Cmd z, voltages are [3, 251, 2, 165]
stored values= [1, 2, 3, 4]
address= 0  answer:  [3, 251, 2, 165]
CPU 1019 Battery 677
CPU 4.98 Battery 14.05 Volts 2017-12-28 17:09:54
... Noreply to cmd 9
Hey, it is here!!!!
!!!!!!!
.......
setting scratchBuf to [True, False, False, True, True, False, False, True]
scratchBuf [True, False, False, True, True, False, False]
Reply: ÿHCM§LJP len 10

The client script is much simpler. Let's call it autogate_client.py. You only use the TCP/IP version:
#---------------------------------------------------------------------------#
# import the various server implementations
#---------------------------------------------------------------------------#
from pymodbus.client.sync import ModbusTcpClient as ModbusClient

You connect to the server thus:
client = ModbusClient('localhost', port=5020)
client.connect()

import sys
from time import localtime, strftime, sleep
if __name__ == "__main__":
    if len(sys.argv) == 2 :
        if sys.argv[1] == 'open' or sys.argv[1] == 'close' \
           or sys.argv[1] == 'Open' or sys.argv[1] == 'Close':
            coils = [True, False, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            print sys.argv[1], 'autogate'
            sleep(1);
            coils = [True, True, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
        elif sys.argv[1] == 'ajar' or sys.argv[1] == 'Ajar':
            coils = [True, False, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            print sys.argv[1], 'autogate'
            sleep(1);
            coils = [True, True, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            sleep(2);
            coils = [True, False, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            print sys.argv[1], 'autogate'
            sleep(1);
            coils = [True, True, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
        else:
            # python rocks! evaluate argument as python expression
            coils = eval(sys.argv[1])
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            print 'writing', coils

        rr = client.read_holding_registers(0,4,unit=0x01)# 2016-12-20
        #print "read_holding_registers 1", rr.registers
        #print "read_holding_registers type", type(rr.registers)
        #print "read_holding_registers list data type", type(rr.registers[0])
        CPU_Voltage = rr.registers[0] * 256 + rr.registers[1]
        Calibrated_CPU_Voltage = (CPU_Voltage * 5.0) / 1024
        Battery_Voltage = rr.registers[2] * 256 + rr.registers[3]
        Calibrated_Battery_Voltage = (Battery_Voltage * 13.53) / 652
        print "read_holding_registers 1", rr.registers, 'CPU', \
              (rr.registers[0] * 256 + rr.registers[1]), 'Battery', \
              rr.registers[2] * 256 + rr.registers[3]
        print 'CPU', "{:4.2f}".format(Calibrated_CPU_Voltage), \
              'Battery', "{:4.2f}".format(Calibrated_Battery_Voltage),\
              'Volts',strftime("%Y-%m-%d %H:%M:%S", localtime())
    else:
        print 'Wrong/no arguments', len(sys.argv)

The main code is:
            coils = [True, False, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
You set the bits you want (only the first 2 are implemented) and use client.write_coils() to write to the Modbus database. The server process does the rest. 

Typical output is:
$python ./autogate_client.py open
open autogate
read_holding_registers 1 [3, 251, 2, 62] CPU 1019 Battery 574
CPU 4.98 Battery 11.91 Volts 2017-12-28 16:14:06

To close the autogate you run the client again:
$python ./autogate_client.py open
open autogate
read_holding_registers 1 [3, 251, 2, 62] CPU 1019 Battery 574
CPU 4.98 Battery 11.91 Volts 2017-12-28 16:31:13

Now when I am on foot I find it useful not to open the gate wide, as a dog might then be tempted to bolt. This opens a man-size opening by triggering the relay twice:
$python ./autogate_client.py ajar
ajar autogate
ajar autogate
read_holding_registers 1 [3, 251, 2, 62] CPU 1019 Battery 574
CPU 4.98 Battery 11.91 Volts 2017-12-28 16:31:07

And that is all there is to it. Now for the PHP script.

$cat  ~/autogate/bluetooth/html/autogate.html 
   <!-- \/ starthtml -->
<html>
 <head>
   <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
   <META NAME="keywords" CONTENT="Heong Chee Meng electronics engineer system sof
tware hardware digital design analog design FPGA VHDL parametric tester Windows N
T realtime device driver Linux kernel hacking semiconductor manufacturing SCADA e
mbedded Seremban Malaysia KM48T02 DAC71">
   <META NAME="description" CONTENT="Heong Chee Meng's Autogate Remote Control website.">
   <META NAME="author" CONTENT="Heong Chee Meng">
   <TITLE>Heong's Autogate Remote Control Website</TITLE>
 </head>
Heong's Autogate Remote Control Website

<p>
<p>
<p>
<p>
<p style="float: left; width: 33.3%; text-align: "center>
<form action="openbutton.php" method="post">
  <button type="submit" name="open" value="Connect"><img width="360" height="360" alt="Connect" src="./open.svg" align="left"></button> 
</form>
<p>
<p>
<p>
<p>
<p style="float: left; width: 33.3%; text-align: center">
<form action="openbutton.php" method="post">
  <button type="submit" name="ajar" value="Connect"><img width="360" height="360" alt="Connect" src="./ajar.svg" align="right"></button>
</form>

<p>
</BODY>
</html>
<!-- /\ end html  -->

And the PHP script is:

$cat  ~/autogate/bluetooth/html/openbutton.php 
<html>
<body>
<article>
<?php
  if (isset($_POST['open'])) {
    $result = shell_exec('python autogate_client.py open');
    echo "Done!<pre>$result</pre>";
  }
  if (isset($_POST['ajar'])) {
    $result = shell_exec('python autogate_client.py ajar');
    echo "Done!<pre>$result</pre>";
  }
  header("Location: ./autogate.html");
?>
</article>
</body>
</html>

Aim your browser (I used Google Chrome) at the webserver: xx.xx.xx.xx/autogate/autogate.html and there you have it- an IoT bluetooth Autogate remote. 

Happy Trails

No comments:

Post a Comment