Interfacing with Retro: Dictionary Extraction

Following up on my prior post where I briefly raised the idea of building a separate interface to Retro, I've decided to do some experiments to see how far I can get. As a major goal, I want to try to do this without requiring a new or modified image.

The first thing of relevance to me is being able to explore the contents of the image. The VM state is easily accessible, but unlike Parable, everything else lives inside a big binary blob. Fortunately most of the layout of this hasn't changed in years, so some things can be reliably located. 

The dictionary is setup as a linked list. The pointer to the most recent node is at offset 2, so we can start there and walk through it to obtain each header. But this still isn't good enough. To make interfacing useful, it'll be helpful to have all of the information in an easier to work with set.

For my prototyping, I'm using Python. Assuming that the image is loaded into an array named memory, a dictionary can be constructed using the following code:

def populate_dictionary(memory):
    dictionary = {}
    last = memory[memory[2]]
    while last != 0:
        name = ''
        i = memory[last] + 4
        while memory[i] != 0:
            name = name + chr(memory[i])
            i += 1
        if memory[last] != 0:
            dictionary[name] = construct_header(memory, memory[last])
        last = memory[last]
    return dictionary

def construct_header(memory, dt):
    header = {}
    header['dt'] = dt
    header['prior'] = memory[dt]
    header['class'] = d_to_class(memory, dt)
    header['xt'] = d_to_xt(memory, dt)
    header['doc'] = d_to_doc(memory, dt)
    header['rawname'] = d_to_name(memory, dt)
    return header

def d_to_class(memory, dt):
    cls = memory[dt + 1]
    return cls

def d_to_doc(memory, dt):
    doc = memory[dt + 3]
    return doc

def d_to_xt(memory, dt):
    xt = memory[dt + 2]
    return xt

def d_to_name(memory, dt):
    name = memory[dt + 4]
    return name

This returns a dictionary with each function name as a key that corresponds to another dictionary with fields for each of the elements in the actual Retro header. Given this, the next step will be modifying the VM to evaluate a specific function.

Contemplations on Retro

My biggest dislike with Retro is the I/O model. Or more specifically, the assumption that the user will interact through a TTY-style interface. Parable has shown that splitting the interface into a separate layer can make it much easier to bring up on different platforms without wasting resources emulating a traditional terminal.

I'm wondering if I can achieve this in Retro, without a full scale rewrite of the VM and image. I think I can. The image format is easy to work with, and there are plenty of hooks where I could effectively duplicate the current listener as a secondary tool, apart from the image. Perhaps it's time to take the first few steps towards this. 

Funeral: A toolchain for creating ePub documents

I spend a lot of time reading. I have purchased over 500 ebooks in the last decade and read well more than that. From time to time I find a book I like a lot, and then I take the time to move it into a plain text format from which I can generate ebooks in whatever format the devices I am using support. At last count I have 95 books, with about ten more candidates waiting for processing and cleanup.

I tried to use Calibre for this, but performance issues proved annoying. Eventually I wrote my own array of tools, which I'm now working on cleaning up and preparing for release. The first of these is Funeral, a set of tools for converting plain text documents into valid HTML, ePub, and azw formats.

Funeral consists of a collection of bash scripts, with some Python and PHP mixed in. I write/store my source texts in ReStructured Text format, so Python is used to support rst2html for the initial conversion. I run the generated HTML through HTML Tidy to clean up errors. Then it gets passed to a PHP script which breaks it apart and reassembles it into a valid ePub. And finally the ePub can be passed to KindleGen for conversion to azw.

The directory structure is laid out in a pretty straightforward manner. The top level directory contains the export script, a scripts directory containing some helper scripts, a build directory which holds the PHP code used to create the ePub, and text, html, epub, and azw directories which hold the books.

New books go in text, under a subdirectory for each author. After editing is complete, they are converted to HTML using the export script:

./export html

Then to epub:

./export epub

And finally, (if we have KindleGen), azw:

./export azw

The biggest frustration I have with this process is that my current export script is an all or none proposition. Exporting just one book isn't possible without manually carrying out the steps the script performs, so I typically wait until I have a batch ready to go. A full cycle (html, epub, azw) with my 95 books takes about 13 minutes.

But getting back to the point; today I'm releasing the source for Funeral. It's now available on github.com/crcx/funeral and can be downloaded/forked/etc. It's under the ISC License.

I have some plans for improving this once I complete a couple of other projects (mainly Apologue and an update to my tea timer). Going forward, I want to remove the PHP code and rewrite the ePub creation code in Python. Then I can use finally start reducing and simplifying the shell scripts (e.g., using PyTidyLib instead of the standalone HTML tidy tool). This should make it easier to integrate into an ebook management tool I'd like to revive as well. I'll be targeting mid-September to begin work on these updates.

Apologue Update: 08.15.2014

I have a few minor updates on Apologue. The first version is basically complete. I'm happy with the overall UI layout, have it working smoothly for my purposes, and have some nice language tweaks that can be toggled as desired. The only thing still lagging behind is documentation. I'm continuing to write this, but it's slow going. I have a little time off this weekend, so I will try to finish up this over the weekend.

There have been a couple of minor adjustments to the interface. The only one of significance is that the stack display now uses background colors to indicate types. I find that this makes it easier for me to quickly group and locate specific groups of results in the output.

Retro 11.6.1 has been released

I've pushed out Retro 11.6.1, a minor update featuring a couple of patches from Marc Simpson. This release does not make any changes to the retroImage. It does fix a bug in the ANS Forth implementation of Ngaro and adds support for treating doc{ }doc pairings as comments under vim.


There will be an 11.6.2 release with a few more changes in the next few weeks. I'm still going through the dissect' vocabulary and autopsy trying to decide on the best means of fixing problems that have been mentioned to me.

Delays in Retro 11.6.1

Delays are always annoying, especially when they are unexpected.

While working on the 11.6.1 release, Marc pointed out a couple of bugs that affect the Autopsy debugger. I'm working to fix these before I finish the 11.6.1 update. If I can't finish them soon, I may still push out 11.6.1 with release notes covering the bug, and then an 11.6.2 once I decide how to proceed with fixing it.