alihan/jupyter_micropython_kernel
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Now able to do %websocketconnect and do it over the webrepl

Browse Source
This commit is contained in:
Julian Todd
2017-10-05 18:00:03 +01:00
parent f9c535e7ac
commit 768b206524
5 changed files with 197 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
README.md
View File

@@ -2,21 +2,6 @@
Jupyter kernel to interact with a MicroPython ESP8266 or ESP32 over its serial REPL.
## Background
This had been proposed as for an enhancement to webrepl with the idea of a jupyter-like
interface to webrepl rather than the faithful emulation of a command line: https://github.com/micropython/webrepl/issues/32
Other known projects that have implemented a Jupyter Micropython kernel are:
* https://github.com/adafruit/jupyter_micropython_kernel
* https://github.com/willingc/circuitpython_kernel
* https://github.com/TDAbboud/mpkernel
* https://github.com/takluyver/ubit_kernel
In my defence, this is not an effect of not-invented-here syndrome; I did not discover most of them until I
had mostly written this one. But for my purposes, this is more robust and contains debugging (of the
serial connections) capability.
## Installation
First install Jupyter: http://jupyter.org/install.html (the Python3 version)
@@ -106,4 +91,48 @@ and then doing
and
%readbytes
## Background
This had been proposed as an enhancement to webrepl with the idea of a jupyter-like
interface to webrepl rather than their faithful emulation of a command line: https://github.com/micropython/webrepl/issues/32
My first implementation operated a spawned-process asyncronous sub-kernel that handled the serial connection.
Ascync technology requires the whole program to work this way, or none of it.
So my next iteration was going to do it using standard python threads to handle the blocking
of the serial connections.
However, further review proved that this was unnecessarily complex if you consider the whole
kernel itself to be operating asyncronously with the front end notebook UI. In particular,
if the notebook can independently issue Ctrl-C KeyboardInterrupt signals into the kernel, there is no longer
a need to worry about what happens when it hangs waiting for input from a serial connection.
Other known projects that have implemented a Jupyter Micropython kernel are:
* https://github.com/adafruit/jupyter_micropython_kernel
* https://github.com/willingc/circuitpython_kernel
* https://github.com/TDAbboud/mpkernel
* https://github.com/takluyver/ubit_kernel
In my defence, this is not an effect of not-invented-here syndrome; I did not discover most of these
other projects until I had mostly written this one.
I do think that for robustness it is important to expose the full processes
of making connections and But for my purposes, this is more robust and contains debugging (of the
serial connections) capability.
Other known projects to have made Jupyter-like or secondary interfaces to Micropython:
* https://github.com/nickzoic/mpy-webpad
* https://github.com/BetaRavener/uPyLoader
The general approach of all of these is to make use of the Ctrl-A
paste mode with its Ctrl-D end of message signals.
The problem with this mode is it was actually designed for
automatic testing rather than supporting an interactive REPL (Read Execute Print Loop) system
(citation required), so there can be reliability issues to do with
accidentally escaping from this mode or not being able to detect the state
of being in it.
For example, you can't safely do a Ctrl-B to leave the paste mode and then a
Ctrl-A to re-enter paste mode cleanly, because a Ctrl-B in the non-paste mode
will reboot the device.

93
jupyter_micropython_kernel/deviceconnector.py
View File

@@ -1,5 +1,6 @@
import logging, sys, time, os, re, base64
import serial, socket, serial.tools.list_ports, select
import websocket # the old non async one
serialtimeout = 0.5
serialtimeoutcount = 10
@@ -12,6 +13,7 @@ def guessserialport():
return sorted([x[0] for x in serial.tools.list_ports.grep("")])
# merge uncoming serial stream and break at OK, \x04, >, \r\n, and long delays
# (must make this a member function so does not have to switch on the type of s)
def yieldserialchunk(s):
res = [ ]
n = 0
@@ -19,12 +21,20 @@ def yieldserialchunk(s):
try:
if type(s) == serial.Serial:
b = s.read()
else:
elif type(s) == socket.socket:
r,w,e = select.select([s], [], [], serialtimeout)
if r:
b = s._sock.recv(1)
else:
b = b''
else: # websocket
r,w,e = select.select([s], [], [], serialtimeout)
if r:
b = s.recv()
if type(b) == str:
b = b.encode("utf8") # handle fact that strings come back from this interface
else:
b = b''
except serial.SerialException as e:
yield b"\r\n**[ys] "
@@ -64,13 +74,24 @@ class DeviceConnector:
def __init__(self, sres):
self.workingserial = None
self.workingsocket = None
self.workingwebsocket = None
self.workingserialchunk = None
self.sres = sres
def workingserialreadall(self): # usually used to clear the incoming buffer
assert self.workingserial is not None
def workingserialreadall(self): # usually used to clear the incoming buffer, results are printed out rather than used
if self.workingserial:
return self.workingserial.read_all()
if self.workingwebsocket:
res = [ ]
while True:
r,w,e = select.select([self.workingwebsocket],[],[],0)
if not r:
break
res.append(self.workingwebsocket.recv())
return "".join(res) # this is returning a text array, not bytes
# though a binary frame can be stipulated according to websocket.ABNF.OPCODE_MAP
# fix this when we see it
# socket case, get it all down
res = [ ]
@@ -92,6 +113,10 @@ class DeviceConnector:
self.sres("Closing socket {}\n".format(str(self.workingsocket)))
self.workingsocket.close()
self.workingsocket = None
if self.workingwebsocket is not None:
self.sres("Closing websocket {}\n".format(str(self.workingwebsocket)))
self.workingwebsocket.close()
self.workingwebsocket = None
def serialconnect(self, portname, baudrate):
self.disconnect()
@@ -143,12 +168,28 @@ class DeviceConnector:
self.sres("Socket ConnectionRefusedError {}".format(str(e)))
def websocketconnect(self, websocketurl):
self.disconnect()
try:
self.workingwebsocket = websocket.create_connection(websocketurl, 5)
self.workingwebsocket.settimeout(serialtimeout)
except socket.timeout:
self.sres("Websocket Timeout after 5 seconds {}\n".format(websocketurl))
except ValueError as e:
self.sres("WebSocket ValueError {}\n".format(str(e)))
except ConnectionResetError as e:
self.sres("WebSocket ConnectionError {}\n".format(str(e)))
except OSError as e:
self.sres("WebSocket OSError {}\n".format(str(e)))
except websocket.WebSocketException as e:
self.sres("WebSocketException {}\n".format(str(e)))
def receivestream(self, bseekokay, bwarnokaypriors=True, b5secondtimeout=False):
n04count = 0
brebootdetected = False
for j in range(2): # for restarting the chunking when interrupted
if self.workingserialchunk is None:
self.workingserialchunk = yieldserialchunk(self.workingserial or self.workingsocket)
self.workingserialchunk = yieldserialchunk(self.workingserial or self.workingsocket or self.workingwebsocket)
indexprevgreaterthansign = -1
index04line = -1
@@ -219,35 +260,36 @@ class DeviceConnector:
break # out of the for loop
def sendtofile(self, destinationfilename, bappend, bbinary, filecontents):
if self.workingserial:
if self.workingserial or self.workingwebsocket:
sswrite = self.workingserial.write if self.workingserial else self.workingwebsocket.send
fmodifier = ("a" if bappend else "w")+("b" if bbinary else "")
if bbinary:
self.workingserial.write(b"import ubinascii; O6 = ubinascii.a2b_base64\r\n")
self.workingserial.write("O=open({}, '{}')\r\n".format(repr(destinationfilename), fmodifier).encode())
sswrite(b"import ubinascii; O6 = ubinascii.a2b_base64\r\n")
sswrite("O=open({}, '{}')\r\n".format(repr(destinationfilename), fmodifier).encode())
if bbinary:
chunksize = 30
for i in range(int(len(filecontents)/chunksize)+1):
bchunk = filecontents[i*chunksize:(i+1)*chunksize]
self.workingserial.write(b'O.write(O6("')
self.workingserial.write(base64.encodebytes(bchunk)[:-1])
self.workingserial.write(b'"))\r\n')
sswrite(b'O.write(O6("')
sswrite(base64.encodebytes(bchunk)[:-1])
sswrite(b'"))\r\n')
if (i%10) == 9:
self.workingserial.write(b'\r\x04') # intermediate executions
sswrite(b'\r\x04') # intermediate executions
self.receivestream(bseekokay=True)
self.sres("{} chunks sent so far\n".format(i+1))
self.sres("{} chunks sent done".format(i+1))
else:
for i, line in enumerate(filecontents.splitlines(True)):
self.workingserial.write("O.write({})\r\n".format(repr(line)).encode())
sswrite("O.write({})\r\n".format(repr(line)).encode())
if (i%10) == 9:
self.workingserial.write(b'\r\x04') # intermediate executions
sswrite(b'\r\x04') # intermediate executions
self.receivestream(bseekokay=True)
self.sres("{} lines sent so far\n".format(i+1))
self.sres("{} lines sent done".format(i+1))
self.workingserial.write("O.close()\r\n".encode())
self.workingserial.write(b'\r\x04')
sswrite("O.close()\r\n".encode())
sswrite(b'\r\x04')
self.receivestream(bseekokay=True)
else:
@@ -255,15 +297,16 @@ class DeviceConnector:
def enterpastemode(self):
# now sort out connection situation
if self.workingserial:
self.workingserial.write(b'\r\x03\x03') # ctrl-C: kill off running programs
if self.workingserial or self.workingwebsocket:
sswrite = self.workingserial.write if self.workingserial else self.workingwebsocket.send
sswrite(b'\r\x03\x03') # ctrl-C: kill off running programs
l = self.workingserialreadall()
if l:
self.sres('[x03x03] ')
self.sres(str(l))
#self.workingserial.write(b'\r\x02') # ctrl-B: leave paste mode if still in it <-- doesn't work as when not in paste mode it reboots the device
self.workingserial.write(b'\r\x01') # ctrl-A: enter raw REPL
self.workingserial.write(b'1\x04') # single character program to run so receivestream works
sswrite(b'\r\x01') # ctrl-A: enter raw REPL
sswrite(b'1\x04') # single character program to run so receivestream works
else:
self.workingsocket.write(b'1\x04') # single character program to run so receivestream works
self.receivestream(bseekokay=True, bwarnokaypriors=False)
@@ -272,6 +315,9 @@ class DeviceConnector:
if self.workingserial:
nbyteswritten = self.workingserial.write(bytestosend)
return ("serial.write {} bytes to {} at baudrate {}".format(nbyteswritten, self.workingserial.port, self.workingserial.baudrate))
elif self.workingwebsocket:
nbyteswritten = self.workingwebsocket.send(bytestosend)
return ("serial.write {} bytes to {}".format(nbyteswritten, "websocket"))
else:
nbyteswritten = self.workingsocket.write(bytestosend)
return ("serial.write {} bytes to {}".format(nbyteswritten, str(self.workingsocket)))
@@ -281,16 +327,23 @@ class DeviceConnector:
self.workingserial.write(b"\x03\r") # quit any running program
self.workingserial.write(b"\x02\r") # exit the paste mode with ctrl-B
self.workingserial.write(b"\x04\r") # soft reboot code
elif self.workingwebsocket:
self.workingwebsocket.send(b"\x03\r") # quit any running program
self.workingwebsocket.send(b"\x02\r") # exit the paste mode with ctrl-B
self.workingwebsocket.send(b"\x04\r") # soft reboot code
def writeline(self, line):
if self.workingserial:
self.workingserial.write(line.encode("utf8"))
self.workingserial.write(b'\r\n')
elif self.workingwebsocket:
self.workingwebsocket.send(line.encode("utf8"))
self.workingwebsocket.send(b'\r\n')
else:
self.workingsocket.write(line.encode("utf8"))
self.workingsocket.write(b'\r\n')
def serialexists(self):
return self.workingserial or self.workingsocket
return self.workingserial or self.workingsocket or self.workingwebsocket

86
jupyter_micropython_kernel/kernel.py
View File

@@ -22,6 +22,11 @@ ap_socketconnect.add_argument('--raw', help='Just open connection', action='stor
ap_socketconnect.add_argument('ipnumber', type=str)
ap_socketconnect.add_argument('portnumber', type=int)
ap_websocketconnect = argparse.ArgumentParser(prog="%websocketconnect", add_help=False)
ap_websocketconnect.add_argument('--raw', help='Just open connection', action='store_true')
ap_websocketconnect.add_argument('websocketurl', type=str, default="ws://192.168.4.1:8266", nargs="?")
ap_websocketconnect.add_argument("--password", type=str)
ap_writebytes = argparse.ArgumentParser(prog="%writebytes", add_help=False)
ap_writebytes.add_argument('-b', help='binary', action='store_true')
ap_writebytes.add_argument('stringtosend', type=str)
@@ -38,8 +43,17 @@ def parseap(ap, percentstringargs1):
except SystemExit: # argparse throws these because it assumes you only want to do the command line
return None # should be a default one
# * sendtofile has -a for append
# * left in buffer not taking account of brebootdetected
# 1. 8266 websocket feature into the main system
# 2. Complete the implementation of websockets on ESP32
# 3. Create the streaming of pulse measurements to a simple javascript frontend and listing
# 4. Try implementing ESP32 webrepl over these websockets using exec()
# 5. Include %magic commands for flashing the ESP firmware (defaulting to website if file not listed)
# 6. Finish debugging the IR codes
# * upgrade picoweb to handle jpg and png and js
# * code that serves a websocket to a browser from picoweb
# then make the websocket from the ESP32 as well
# then make one that serves out sensor data just automatically
@@ -50,6 +64,30 @@ def parseap(ap, percentstringargs1):
# * record incoming bytes (eg when in enterpastemode) that haven't been printed
# and print them when there is Ctrl-C
# the socket to ESP32 method could either run exec, or
# save to a file, import it, then delete the modele from sys.modules[]
# * potentially run commands to commission the ESP
# esptool.py --port /dev/ttyUSB0 erase_flash
# esptool.py --port /dev/ttyUSB0 --baud 460800 write_flash --flash_size=detect 0 binaries/esp8266-20170108-v1.8.7.bin --flash_mode dio
# esptool.py --chip esp32 --port /dev/ttyUSB0 write_flash -z 0x1000 esp32...
# It looks like we can handle the 8266 webrepl in the following way:
# import websocket # note no s, so not the asyncio one
#ws = websocket.create_connection("ws://192.168.4.1:8266/")
#result = ws.recv()
#if result == 'Password: ':
# ws.send("wpass\r\n")
#print("Received '%s'" % result)
#ws.close()
# and then treat like the serial port
# should also handle shell-scripting other commands, like arpscan for mac address to get to ip-numbers
# compress the websocket down to a single straightforward set of code
# take 1-second of data (100 bytes) and time the release of this string
# to the web-browser
class MicroPythonKernel(Kernel):
implementation = 'micropython_kernel'
@@ -93,6 +131,26 @@ class MicroPythonKernel(Kernel):
# self.dc.enterpastemode()
return None
if percentcommand == ap_websocketconnect.prog:
apargs = parseap(ap_websocketconnect, percentstringargs[1:])
self.dc.websocketconnect(apargs.websocketurl)
if self.dc.workingwebsocket:
self.sres("\n ** WebSocket connected **\n\n", 32)
self.sres(str(self.dc.workingwebsocket))
self.sres("\n")
if not apargs.raw:
pline = self.dc.workingwebsocket.recv()
self.sres(pline)
if pline == 'Password: ' and apargs.password is not None:
self.dc.workingwebsocket.send(apargs.password)
self.dc.workingwebsocket.send("\r\n")
res = self.dc.workingserialreadall()
self.sres(res) # '\r\nWebREPL connected\r\n>>> '
if not apargs.raw:
self.dc.enterpastemode()
return None
if percentcommand == "%lsmagic":
self.sres("%disconnect\n disconnects serial\n\n")
self.sres("%lsmagic\n list magic commands\n\n")
@@ -106,6 +164,9 @@ class MicroPythonKernel(Kernel):
self.sres(" connects to a socket of a device over wifi\n\n")
self.sres("%suppressendcode\n doesn't send x04 or wait to read after sending the cell\n")
self.sres(" (assists for debugging using %writebytes and %readbytes)\n\n")
self.sres(re.sub("usage: ", "", ap_websocketconnect.format_usage()))
self.sres(" connects to the webREPL websocket of an ESP8266 over wifi\n")
self.sres(" websocketurl defaults to ws://192.168.4.1:8266 but be sure to be connected\n\n")
self.sres(re.sub("usage: ", "", ap_writebytes.format_usage()))
self.sres(" does serial.write() of the python quoted string given\n\n")
return None
@@ -122,6 +183,7 @@ class MicroPythonKernel(Kernel):
apargs = parseap(ap_writebytes, percentstringargs[1:])
bytestosend = apargs.stringtosend.encode().decode("unicode_escape").encode()
self.sres(self.dc.writebytes(bytestosend))
return None
if percentcommand == "%readbytes":
l = self.dc.workingserialreadall()
@@ -137,6 +199,10 @@ class MicroPythonKernel(Kernel):
self.sres("Did you mean %rebootdevice?\n", 31)
return None
if percentcommand == "%sendbytes":
self.sres("Did you mean %writebytes?\n", 31)
return None
if percentcommand == "%reboot":
self.sres("Did you mean %rebootdevice?\n", 31)
return None
@@ -215,7 +281,9 @@ class MicroPythonKernel(Kernel):
return {'status': 'ok', 'execution_count': self.execution_count, 'payload': [], 'user_expressions': {}}
interrupted = False
if self.dc.serialexists():
# clear buffer out before executing any commands (except the readbytes one)
if self.dc.serialexists() and not re.match("%readbytes", code):
priorbuffer = None
try:
priorbuffer = self.dc.workingserialreadall()
@@ -227,15 +295,17 @@ class MicroPythonKernel(Kernel):
self.sres("You may need to reconnect")
if priorbuffer:
for pbline in priorbuffer.splitlines():
if type(priorbuffer) == bytes:
try:
ur = pbline.decode()
priorbuffer = priorbuffer.decode()
except UnicodeDecodeError:
ur = str(pbline)
if deviceconnector.wifimessageignore.match(ur):
priorbuffer = str(priorbuffer)
for pbline in priorbuffer.splitlines():
if deviceconnector.wifimessageignore.match(pbline):
continue # filter out boring wifi status messages
self.sres('[leftinbuffer] ')
self.sres(str([ur]))
self.sres(str([pbline]))
self.sres('\n')
try:

2
micropython_webserve.ipynb
View File

@@ -483,7 +483,7 @@
}
],
"source": [
"upip.install(\"utemplate\")"
"upip.install(\"utemplate\")\n"
]
},
{

2
setup.py
View File

@@ -9,6 +9,6 @@ setup(name='jupyter_micropython_kernel',
url='https://github.com/goatchurchprime/jupyter_micropython_kernel',
license='GPL3',
packages=['jupyter_micropython_kernel'],
install_requires=['pyserial']
install_requires=['pyserial', 'websocket']
)