Interfacing with Retro: Executing Functions While Preserving the Stack

When I left off, I had finished an initial function for executing a single function. This worked, but didn't allow me to carry the stack contents between runs. It also suffered from not supporting class handlers, which are one of the major features of Retro.

So going forward, I needed to fix both of these. I implemented a new routine (executeWithClass) which takes a tuple containing an xt and a class, as well as the stack, memory, and i/o handlers.

Constructing the tuple is handled by a small helper function:

def lookup(name, d):
    return d[name]['xt'], d[name]['class']

The executeWithClass is pretty straightfoward. Instead of creating a new stack, we pass one in along with the tuple returned by lookup.

def executeWithClass( pair, stack, memory, inputs ):
  global EXIT
  ip = pair[1]
  stack.append(pair[0])
  EXIT = len( memory )
  address = [] * 1024
  ports = [0] * 12
  files = [0] * 8

  while ip < EXIT:
    opcode = memory[ip]

    # There are 31 opcodes ( 0 .. 30 ).
    # Instructions above this range are treated as implicit calls.

    if opcode > 30:
      address.append( ip )
      ip = memory[ip] - 1
      try:
        while memory[ip + 1] == 0:
          ip += 1
      except IndexError: pass

    else:

      if   opcode ==  0:   # nop
        pass

      elif opcode ==  1:   # lit
        ip += 1
        stack.append( memory[ip] )

      elif opcode ==  2:   # dup
        stack.append( stack[-1] )

      elif opcode ==  3:   # drop
        stack.pop()

      elif opcode ==  4:   # swap
        a = stack[-2]
        stack[-2] = stack[-1]
        stack[-1] = a

      elif opcode ==  5:   # push
        address.append( stack.pop() )

      elif opcode ==  6:   # pop
        stack.append( address.pop() )

      elif opcode ==  7:   # loop
        stack[-1] -= 1
        if stack[-1] != 0 and stack[-1] > -1:
          ip += 1
          ip = memory[ip] - 1
        else:
          ip += 1
          stack.pop()

      elif opcode ==  8:   # jump
        ip += 1
        ip = memory[ip] - 1
        if memory[ip + 1] == 0:
          ip += 1
          if memory[ip + 1] == 0:
            ip += 1

      elif opcode ==  9:   # return
        ip = address.pop()
        if memory[ip + 1] == 0:
          ip += 1
          if memory[ip + 1] == 0:
            ip += 1

      elif opcode == 10:   # >= jump
        ip += 1
        a = stack.pop()
        b = stack.pop()
        if b > a:
          ip = memory[ip] - 1

      elif opcode == 11:   # <= jump
        ip += 1
        a = stack.pop()
        b = stack.pop()
        if b < a:
          ip = memory[ip] - 1

      elif opcode == 12:   # != jump
        ip += 1
        a = stack.pop()
        b = stack.pop()
        if b != a:
          ip = memory[ip] - 1

      elif opcode == 13:   # == jump
        ip += 1
        a = stack.pop()
        b = stack.pop()
        if b == a:
          ip = memory[ip] - 1

      elif opcode == 14:   # @
        stack[-1] = memory[stack[-1]]

      elif opcode == 15:   # !
        mi = stack.pop()
        memory[mi] = stack.pop()

      elif opcode == 16:   # +
        t = stack.pop()
        stack[ -1 ] += t
        stack[-1] = unpack('=l', pack('=L', stack[-1] & 0xffffffff))[0]

      elif opcode == 17:   # -
        t = stack.pop()
        stack[-1] -= t
        stack[-1] = unpack('=l', pack('=L', stack[-1] & 0xffffffff))[0]

      elif opcode == 18:   # *
        t = stack.pop()
        stack[-1] *= t
        stack[-1] = unpack('=l', pack('=L', stack[-1] & 0xffffffff))[0]

      elif opcode == 19:   # /mod
        a = stack[-1]
        b = stack[-2]
        stack[-1], stack[-2] = rxDivMod( b, a )
        stack[-1] = unpack('=l', pack('=L', stack[-1] & 0xffffffff))[0]
        stack[-2] = unpack('=l', pack('=L', stack[-2] & 0xffffffff))[0]

      elif opcode == 20:   # and
        t = stack.pop()
        stack[-1] &= t

      elif opcode == 21:   # or
        t = stack.pop()
        stack[-1] |= t

      elif opcode == 22:   # xor
        t = stack.pop()
        stack[-1] ^= t

      elif opcode == 23:   # <<
        t = stack.pop()
        stack[-1] <<= t

      elif opcode == 24:   # >>
        t = stack.pop()
        stack[-1] >>= t

      elif opcode == 25:   # 0;
        if stack[-1] == 0:
          stack.pop()
          ip = address.pop()

      elif opcode == 26:   # inc
        stack[-1] += 1

      elif opcode == 27:   # dec
        stack[-1] -= 1

      elif opcode == 28:   # in
        t = stack[-1]
        stack[-1] = ports[t]
        ports[t] = 0

      elif opcode == 29:   # out
        pi = stack.pop()
        ports[ pi ] = stack.pop()

      elif opcode == 30:   # wait
        # Only call if we have pending I/O
        if ports[0] == 0:
          ip = rxHandleDevices( ip, stack, address, ports, memory, files, inputs )

    ip += 1
  return stack, address, memory

And now it's functional. Things like this are now possible:

  d = populate_dictionary(memory)
  stack = [] * 128
  stack.append(1)
  stack.append(2)
  try:
    executeWithClass(lookup('+', d), stack, memory, inputs)
  except:
    pass
  try:
    executeWithClass(lookup('putn', d), stack, memory, inputs)
  except:
    pass

I think the next thing to do is to implement a parser and build an interpreter loop. And then refine the functions I've written into a usable target.

Interfacing with Retro: Running Functions Directly

Previously I posted some code for constructing a Python dictionary that provided access to the named functions in a retroImage. The next step is being able to run an arbitrary function apart from the listener loop. 

I have achieved this through a dedicated processing loop. The loop is a minor variant on the process loop in retro.py. It has been extended to take a function pointer to start at. With this function, it's now possible to directly execute a function or two:

  d = populate_dictionary(memory)
  try:
      processFunction(d['words']['xt'], memory, inputs)
  except:
      pass

The processFunction routine is currently defined as:

def processFunction( xt, memory, inputs ):
  ip = xt
  global EXIT
  EXIT = len( memory )
  stack = [] * 128
  address = [] * 1024
  ports = [0] * 12
  files = [0] * 8

  while ip < EXIT:
    opcode = memory[ip]

    # There are 31 opcodes ( 0 .. 30 ).
    # Instructions above this range are treated as implicit calls.

    if opcode > 30:
      address.append( ip )
      ip = memory[ip] - 1
      try:
        while memory[ip + 1] == 0:
          ip += 1
      except IndexError: pass

    else:

      if   opcode ==  0:   # nop
        pass

      elif opcode ==  1:   # lit
        ip += 1
        stack.append( memory[ip] )

      elif opcode ==  2:   # dup
        stack.append( stack[-1] )

      elif opcode ==  3:   # drop
        stack.pop()

      elif opcode ==  4:   # swap
        a = stack[-2]
        stack[-2] = stack[-1]
        stack[-1] = a

      elif opcode ==  5:   # push
        address.append( stack.pop() )

      elif opcode ==  6:   # pop
        stack.append( address.pop() )

      elif opcode ==  7:   # loop
        stack[-1] -= 1
        if stack[-1] != 0 and stack[-1] > -1:
          ip += 1
          ip = memory[ip] - 1
        else:
          ip += 1
          stack.pop()

      elif opcode ==  8:   # jump
        ip += 1
        ip = memory[ip] - 1
        if memory[ip + 1] == 0:
          ip += 1
          if memory[ip + 1] == 0:
            ip += 1

      elif opcode ==  9:   # return
        ip = address.pop()
        if memory[ip + 1] == 0:
          ip += 1
          if memory[ip + 1] == 0:
            ip += 1

      elif opcode == 10:   # >= jump
        ip += 1
        a = stack.pop()
        b = stack.pop()
        if b > a:
          ip = memory[ip] - 1

      elif opcode == 11:   # <= jump
        ip += 1
        a = stack.pop()
        b = stack.pop()
        if b < a:
          ip = memory[ip] - 1

      elif opcode == 12:   # != jump
        ip += 1
        a = stack.pop()
        b = stack.pop()
        if b != a:
          ip = memory[ip] - 1

      elif opcode == 13:   # == jump
        ip += 1
        a = stack.pop()
        b = stack.pop()
        if b == a:
          ip = memory[ip] - 1

      elif opcode == 14:   # @
        stack[-1] = memory[stack[-1]]

      elif opcode == 15:   # !
        mi = stack.pop()
        memory[mi] = stack.pop()

      elif opcode == 16:   # +
        t = stack.pop()
        stack[ -1 ] += t
        stack[-1] = unpack('=l', pack('=L', stack[-1] & 0xffffffff))[0]

      elif opcode == 17:   # -
        t = stack.pop()
        stack[-1] -= t
        stack[-1] = unpack('=l', pack('=L', stack[-1] & 0xffffffff))[0]

      elif opcode == 18:   # *
        t = stack.pop()
        stack[-1] *= t
        stack[-1] = unpack('=l', pack('=L', stack[-1] & 0xffffffff))[0]

      elif opcode == 19:   # /mod
        a = stack[-1]
        b = stack[-2]
        stack[-1], stack[-2] = rxDivMod( b, a )
        stack[-1] = unpack('=l', pack('=L', stack[-1] & 0xffffffff))[0]
        stack[-2] = unpack('=l', pack('=L', stack[-2] & 0xffffffff))[0]

      elif opcode == 20:   # and
        t = stack.pop()
        stack[-1] &= t

      elif opcode == 21:   # or
        t = stack.pop()
        stack[-1] |= t

      elif opcode == 22:   # xor
        t = stack.pop()
        stack[-1] ^= t

      elif opcode == 23:   # <<
        t = stack.pop()
        stack[-1] <<= t

      elif opcode == 24:   # >>
        t = stack.pop()
        stack[-1] >>= t

      elif opcode == 25:   # 0;
        if stack[-1] == 0:
          stack.pop()
          ip = address.pop()

      elif opcode == 26:   # inc
        stack[-1] += 1

      elif opcode == 27:   # dec
        stack[-1] -= 1

      elif opcode == 28:   # in
        t = stack[-1]
        stack[-1] = ports[t]
        ports[t] = 0

      elif opcode == 29:   # out
        pi = stack.pop()
        ports[ pi ] = stack.pop()

      elif opcode == 30:   # wait
        # Only call if we have pending I/O
        if ports[0] == 0:
          ip = rxHandleDevices( ip, stack, address, ports, memory, files, inputs )

    ip += 1
  return stack, address, memory

The next steps will be the ability to persist the data stack between calls and execute functions via their associated class handlers. This should open things up quite a bit more. With these it'll be feasible to implement a high level token-by-token interpreter routine, though handling the compiler and Retro's parsing/input routines will require a bit more work.

Interfacing with Retro: Dictionary Extraction

Following up on my prior post where I briefly raised the idea of building a separate interface to Retro, I've decided to do some experiments to see how far I can get. As a major goal, I want to try to do this without requiring a new or modified image.

The first thing of relevance to me is being able to explore the contents of the image. The VM state is easily accessible, but unlike Parable, everything else lives inside a big binary blob. Fortunately most of the layout of this hasn't changed in years, so some things can be reliably located. 

The dictionary is setup as a linked list. The pointer to the most recent node is at offset 2, so we can start there and walk through it to obtain each header. But this still isn't good enough. To make interfacing useful, it'll be helpful to have all of the information in an easier to work with set.

For my prototyping, I'm using Python. Assuming that the image is loaded into an array named memory, a dictionary can be constructed using the following code:

def populate_dictionary(memory):
    dictionary = {}
    last = memory[memory[2]]
    while last != 0:
        name = ''
        i = memory[last] + 4
        while memory[i] != 0:
            name = name + chr(memory[i])
            i += 1
        if memory[last] != 0:
            dictionary[name] = construct_header(memory, memory[last])
        last = memory[last]
    return dictionary

def construct_header(memory, dt):
    header = {}
    header['dt'] = dt
    header['prior'] = memory[dt]
    header['class'] = d_to_class(memory, dt)
    header['xt'] = d_to_xt(memory, dt)
    header['doc'] = d_to_doc(memory, dt)
    header['rawname'] = d_to_name(memory, dt)
    return header

def d_to_class(memory, dt):
    cls = memory[dt + 1]
    return cls

def d_to_doc(memory, dt):
    doc = memory[dt + 3]
    return doc

def d_to_xt(memory, dt):
    xt = memory[dt + 2]
    return xt

def d_to_name(memory, dt):
    name = memory[dt + 4]
    return name

This returns a dictionary with each function name as a key that corresponds to another dictionary with fields for each of the elements in the actual Retro header. Given this, the next step will be modifying the VM to evaluate a specific function.

Contemplations on Retro

My biggest dislike with Retro is the I/O model. Or more specifically, the assumption that the user will interact through a TTY-style interface. Parable has shown that splitting the interface into a separate layer can make it much easier to bring up on different platforms without wasting resources emulating a traditional terminal.

I'm wondering if I can achieve this in Retro, without a full scale rewrite of the VM and image. I think I can. The image format is easy to work with, and there are plenty of hooks where I could effectively duplicate the current listener as a secondary tool, apart from the image. Perhaps it's time to take the first few steps towards this. 

Funeral: A toolchain for creating ePub documents

I spend a lot of time reading. I have purchased over 500 ebooks in the last decade and read well more than that. From time to time I find a book I like a lot, and then I take the time to move it into a plain text format from which I can generate ebooks in whatever format the devices I am using support. At last count I have 95 books, with about ten more candidates waiting for processing and cleanup.

I tried to use Calibre for this, but performance issues proved annoying. Eventually I wrote my own array of tools, which I'm now working on cleaning up and preparing for release. The first of these is Funeral, a set of tools for converting plain text documents into valid HTML, ePub, and azw formats.

Funeral consists of a collection of bash scripts, with some Python and PHP mixed in. I write/store my source texts in ReStructured Text format, so Python is used to support rst2html for the initial conversion. I run the generated HTML through HTML Tidy to clean up errors. Then it gets passed to a PHP script which breaks it apart and reassembles it into a valid ePub. And finally the ePub can be passed to KindleGen for conversion to azw.

The directory structure is laid out in a pretty straightforward manner. The top level directory contains the export script, a scripts directory containing some helper scripts, a build directory which holds the PHP code used to create the ePub, and text, html, epub, and azw directories which hold the books.

New books go in text, under a subdirectory for each author. After editing is complete, they are converted to HTML using the export script:

./export html

Then to epub:

./export epub

And finally, (if we have KindleGen), azw:

./export azw

The biggest frustration I have with this process is that my current export script is an all or none proposition. Exporting just one book isn't possible without manually carrying out the steps the script performs, so I typically wait until I have a batch ready to go. A full cycle (html, epub, azw) with my 95 books takes about 13 minutes.

But getting back to the point; today I'm releasing the source for Funeral. It's now available on github.com/crcx/funeral and can be downloaded/forked/etc. It's under the ISC License.

I have some plans for improving this once I complete a couple of other projects (mainly Apologue and an update to my tea timer). Going forward, I want to remove the PHP code and rewrite the ePub creation code in Python. Then I can use finally start reducing and simplifying the shell scripts (e.g., using PyTidyLib instead of the standalone HTML tidy tool). This should make it easier to integrate into an ebook management tool I'd like to revive as well. I'll be targeting mid-September to begin work on these updates.

Apologue Update: 08.15.2014

I have a few minor updates on Apologue. The first version is basically complete. I'm happy with the overall UI layout, have it working smoothly for my purposes, and have some nice language tweaks that can be toggled as desired. The only thing still lagging behind is documentation. I'm continuing to write this, but it's slow going. I have a little time off this weekend, so I will try to finish up this over the weekend.

There have been a couple of minor adjustments to the interface. The only one of significance is that the stack display now uses background colors to indicate types. I find that this makes it easier for me to quickly group and locate specific groups of results in the output.