Closing The Loop Early For Self-Improvable Graphical Systems

2021-08-17T14:11:12+00:00

Author information

(Redacted to assist reviewers)

The dream: self-improvable systems based on structured representations

My research interest is to combine two properties that, so far, only exist independently.

The first, self-improvability, is the ability for a system to present itself as open for modification to a deep level. The quintissential example is the “Combined Object Lambda Architecture” research by Piumarta et al. However, Smalltalk, Lisp, and Unix as a whole also exhibit this property. Note that, although software within Unix can be modified by changing the source code and rebuilding, the language and abstractions used in the source code are often different to those in the built program. So a Unix-based system as a whole is improved by means of itself, yet its individual components are not.

The second, structuredness, means that data (and code) is represented as graphs and trees instead of strings wherever possible. Examples include Subtext and spreadsheets. This makes parsing and serialising unnecessary, while syntax errors and “tabs versus spaces” are appropriately banished. Code and data can be presented in a variety of context-appropriate ways, taking full advantage of a 2D display, and can be manipulated directly as structures instead of characters. Real graphics, colours and shapes – not just ASCII art – can be included as desired. None of this precludes “typing out” structures on the keyboard, either.

Existing systems that demonstrate self-improvability seem invested in the text-and-parsing world, and systems not in this world do not exhibit self-improvability. I want to combine the best of both.

Alas, while the path for building self-improvable languages is well trodden, its techniques do not clearly translate over to the world of interactive structure. What I am presenting here is my progress so far at navigating these uncharted waters.

Building up from a human-friendly low level

The way to build a self-improvable system is to bootstrap it from some given starting point. An initial substrate must be decided upon. First, we have to decide the form of data, i.e. state, in the substrate. Then, we must ensure that changes to the state can be described in the state itself. In other words, code must be data. With this in place, code can be generated by programs, and the ladder of abstraction can be climbed from Assembler to high-level code descriptions. In other words, we can “close the loop”.

This already happened at the machine level of computing, of course.

The form of state is simply bit-patterns, equivalently numbers, arranged in a flat line and addressed by numbers.
Code is data: machine instructions are encoded as bits (numbers).

The problem with this is that it is human-repellent. You can’t survive in this environment for long without introducing symbolic names in some form - whether encoded as Unicode in the state, or on a piece of paper, or simply in your head. We simply do not identify things by opaque numbers, and nor should we.

Additionally, the state is flat. You can’t insert or grow something without physically moving other stuff to make space. Anything that can be called a structure (trees, graphs etc.) has to be faked as scattered memory blocks pointing to each other.

Historically, this was an unfortunate necessity. But nowadays, we have the opportunity to forget about this level, and instead build things on top of a human-friendly low level:

State is a tree (alternatively, graph) of textual names (which can include spaces!)
Changes to the state (code) are simply pieces of state with a certain format.

It might look like this:

The devil’s in the details

The above description of our starting substrate is a sketch, far from complete. My implementation technology is JavaScript in the web browser, and I know I want an expanding/collapsing tree view as a minimal way to see the current state. As soon as I decide to do something, it’s apparent that there are still many undefined design details. For the “state” half of the problem:

What JavaScript data structures will model the state? (Plain JavaScript objects, a.k.a. dictionaries or maps)
Are we going with strict trees, DAGs or general graphs? (Support general graphs in a tree view.)
How smart should the state be? Should it be “reactive”, notifying listeners when pieces change? (No.)
How will we represent lists? Will they start at 0 or 1? (Lists are just dictionaries with numerical keys, starting at 1.)
Which parts of the state will have special meaning, and which will be user-defined?

And so on. The last question relates to the model of computation, i.e. the “changes to state” half of the problem:

What programming model will lie at the lowest level? (Imperative Assembler.)
How will instructions be arranged? Will each have a “next” pointer? (No, just as a list.)
What format will instructions have? (An “op” field to dispatch the correct behaviour, and any custom arguments.)
What types of instructions will be available? How many? Sophisticated or primitive? (A handful, all primitive.)

Answering these design questions is more art than science. In my case they depended on factors too complicated and boring to justify here, including a heavy dose of personal taste. However, there does flow through a common theme, for which my answers were informed by a principle, or heuristic. The theme is this: should we make the starting substrate more, or less, feature-rich; sophisticated; helpful; smart?

My heuristic is no – our starting substrate should be small, primitive, and quick to implement. That way, the sooner we can leave JavaScript, to start working entirely in-system.

Minimal substrates for maximal progress?

Truly, it is tedious to work with a programming system that only has the bare essentials. It is tempting to see this looming ahead, and to get to work implementing a substrate that is smart, helpful, reactive, whatever – perhaps it even has a standard library. This is a fine goal to have for the system, but such progress must be accomplished in-system, or it is wasted.

It is wasted because it involves putting all our work into code that exists in our JavaScript files, which are inaccessible to the system. This means that we cannot take advantage of anything we invent in the system – visualisations, abstractions, domain-specific languages – to affect these parts of it. They are permanently locked out of self-improvability! Even if we give our system access to the JavaScript, reintroducing these text strings! would corrupt our beautiful world with the ugly anachronisms of parsing and serialising once again, and we might as well give up.

We proceed with the knowledge that any fancy features we implement in JavaScript are temporary crutches that we’ll need to re-implement in-system, sooner or later. In short, the JavaScript is a ladder to throw away once transcended. It can’t quite be “jettisoned without remorse” as Piumarta does with his C++; it’s all we’ve got to “run” anything (short of WebAssembly). However, we can aim for a minimal JavaScript kernel that can, perhaps, be “forgotten without remorse”.

This heuristic, if accepted, leads quite inevitably to an answer to one of the major questions: the base programming model must resemble imperative Assembler. This is because, being the quickest to implement, it will get us out of JavaScript the fastest.

Demo: camera panning lifted out of JavaScript

The following video shows a tree view of system state on the right, next to a 3D rendering window. The camera can be panned around by dragging with the mouse and zoomed by scrolling. This behaviour is running in JavaScript, but notice the flag dragging_in_system in the tree view. When we set it to true, all that happens when we drag is that the press and release positions are recorded under the pointer key. We can then run the instructions to see the very same camera-moving behaviour running in-system. In the next section we will explain the instruction set, and later we will take a closer look at the code in the video.

The basic instruction set

Our reference point is the global root of the state tree. Its immediate children we call registers. A handful of registers are treated specially, but the rest can be named at will and used as local variables by user programs. The special registers are:

next_instruction: this is the Instruction Pointer. It can be pointed to any list and will advance within it. To jump, just overwrite it!
focus: the current focus of attention. This is the bottleneck for load/store operations. It also holds the key when indexing maps (a.k.a. dictionaries).
map: whatever map is held here will be indexed or written to by index and store instructions.
source: this holds the source value for map write operations.

The minimal necessary instructions are as follows:

load <value>: makes focus hold the given immediate value.
deref: treats focus as naming another register, and replaces focus with its contents.
index: uses the focus as a key to look up in map, and replaces map with the corresponding value. (If the map does not have that key, it returns the value of the “default” _ key instead; this is how we get “else” in conditional branches.)
store <register>: copies the contents of focus to the named register.
store: without any register argument, uses the focus as a key to look up in map, and copies the source to this location.
js <func>: executes the given JavaScript function. This is a last-resort window into the world “underneath” and is necessary for making use of Web libraries.

Because these are so primitive, affecting minimal state, they are perhaps best comparable to micro-operations in a CISC processor. What seem like the most primitive possible operations (e.g. copying one piece of state elsewhere, or taking a conditional branch) expand to several or even many instructions. Here are some examples of recurring patterns (heralding future macros and subroutines):

Copying one register to another, reg_dest <- reg_source, expands to load reg_source; deref; store reg_dest.
Writing to maps via registers, reg_map.[reg_key] <- reg_source involves copying the three registers to map, focus and source respectively. In full, this expands to load reg_map; deref; store map; load reg_source; deref; store source; load reg_key; deref; store.
Accessing a fixed path involves repeated load and index pairs for each key. focus <- map.a.b.c expands to load a; index; load b; index; load c; index.

Demo: writing to the tree

The following video shows an initial state with a data structure, lisp_stuff, which is a prototype of what a high-level language structure might look like in this substrate. The next_instruction has been pointed to the instructions register, starting at key 1. The instructions change the value at the path lisp_stuff.args_e.value.args_e.body_e.1.type to the value foobar.

Demo: conditional jump

The initial state here involves the weather, conclusion and finished registers. If the weather is cold the conclusion should be it's cold; if it’s warm the conclusion should be it's warm, and otherwise it's neither!. The branching structure can be seen in the instruction list. Regardless of the branch, they should all merge afterward to set finished to true.

There is no need for a comparison instruction, because there is already an instruction for indexing maps based on a key name. Notice how verbose it is to overwrite next_instruction to make a jump! We need to reset the key to 1 as well as point to a new map, but we cannot do these as two separate instructions or it will jump to the wrong place! Instead, we must prepare a new value to write in one single instruction.

And Now, The Shapes

In the previous two demos, we’ve been using plain HTML and CSS for graphics. For things like arrows and more complicated shapes, we need something else. In a previous project, I used SVG. I discovered that SVG is very unpleasant to work with programmatically. Pain points include the fact that everything is “stringly typed” (structure, even matrix transformations, is encoded as attribute strings) and that one must use tree node ordering to control which shapes go in front of or behind others. This time, I am using THREE.js, a 3D graphics library that is nevertheless a wise choice for serious 2D graphics.

The obvious path forward is to make a bunch of THREE.js API calls in js instructions; however, in this case it pays to consider another way. Think of Commodore 64 BASIC and memory-mapped I/O. In the flat, binary model of state, there is a region of memory known as the framebuffer; you can write values there and see coloured pixels light up on the display.

In keeping with our theme here, how can we lift the flat and binary idea of a “framebuffer” into minimal basic human-friendliness? There are several choices, one being to simply preserve pixel access in a nicer way, but how about this: the framebuffer does for raster graphics what the tree does for vector graphics. Memory-mapped graphics at the machine level, the framebuffer, matches the flat nature of the rest of memory. Why not define memory-mapped graphics at the structured level as structures representing shapes, i.e. as retained-mode vector graphics?

In our case, we are fortunate to already have a tree structure in THREE.js which can be straightforwardly mirrored into our system state. At the time of writing, this is still mostly not implemented. We can, however, return to the camera panning demo from earlier for a closer look.

Demo: camera panning internals

As soon as we have a means for graphics in our substrate, an essential task is to allow zooming and panning around the world. The physical screen can only hold so much, and this task amounts to freeing ourselves from its constraints. Let’s see how panning is achieved in the system’s instruction set:

This relies on the new registers vec_from and vec_to, and the new instruction vsub which returns the vector from vec_from to vec_to. Notice also the format of vectors, particularly that each carries the name of its basis frame. Representing a screen vector in world co-ordinates is as simple as calling an in world instruction.

We have missed no opportunity to put the immensely fundamental infrastructure of vector transformations on a “minimally human-friendly” footing. All too often in programming – even mathematics and physics – we are presented with a vision of “vectors” as mere lists of numbers, and we must keep track of bases and transformations on paper. Worse, even the components are listed as 0, 1, 2 or x, y, z (which are no less meaningless) and we must waste cognition remembering which directions correspond to which indexes. As an exception to the “minimalism” heuristic, it is plausibly worth building “smart” vector and transformation infrastructure directly into the substrate.

The story so far, and ahead

So far, here is the rough sequence of steps involved in bootstrapping a self-improvable graphical system:

Choose an Implementation Substrate IS (JavaScript)
Choose the in-system state format (trees)
Derive the minimal set of instructions to represent computation (load, store, deref, index, js)
Implement (in the IS) the fetch-decode-execute cycle for these instructions (this should be quick)
Implement (in the system) zoom / pan round an unlimited space

We are in the middle of step 5. It is not obvious what the next priority is, but certainly future steps will include the following.

Macros, subroutines and high-level code. Develop calling conventions for re-using chunks of instructions and returning from them; allow unknown “instructions” to be processed by user-defined macro expanders. Use the linear world of Assembler to escape to a structured, Lisp-like expression of behaviour built on maps instead of lists (Masp?)

Render the trees in THREE.js. It was convenient to have a tree view in HTML as the initial, throwaway viewer for system state. However, if the THREE.js view is supposed to be our window into the world, then one day the tree view must be part of it. Appropriate, domain-specific, high-level structures representing tree view nodes can be progressively expanded into shape structures. The subtle art of deciding which responsibilities belong in the substrate, and which belong in JavaScript, is particularly important here.

Innovate visual editing. The tree representation of code will inevitably be verbose and tedious; at least we can try the full spectrum of textual-graphical possibilities here, which cannot be said for text editors on the outside.

Food for thought

My current goal is to see merely an example of a self-improvable graphical system. Beyond this, I hope that the exercise will result in techniques transferrable to other implementation technologies besides JavaScript, and to different answers to the various design questions.

Hypothesis: given a particular way of organising state, a minimal Assembler language of changes to that state can be mechanically derived. Imagine deriving the Assembler language for the following state models: SVG trees in the browser DOM; The Unix filesystem; a general graph structure; 2D continuous space(!)

Hypothesis: there is a general bootstrapping technique, such that the starting point can be any language or platform according to taste or whim. It can be treated as an axiom; given such an implementation technology, the main steps for bootstrapping a self-improvable system can be straightforwardly followed.

References

Ian Piumarta “Accessible Language-Based Environments of Recursive Theories”, Viewpoints Research Institute, 2006
Jonathan Edwards “Subtext: uncovering the simplicity of programming”, ongoing

Welcome to Jekyll!

2021-08-17T14:11:12+00:00

You’ll find this post in your _posts directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run jekyll serve, which launches a web server and auto-regenerates your site when a file is updated.

Jekyll requires blog post files to be named according to the following format:

YEAR-MONTH-DAY-title.MARKUP

Where YEAR is a four-digit number, MONTH and DAY are both two-digit numbers, and MARKUP is the file extension representing the format used in the file. After that, include the necessary front matter. Take a look at the source for this post to get an idea about how it works.

Jekyll also offers powerful support for code snippets:

def print_hi(name)
  puts "Hi, #{name}"
end
print_hi('Tom')
#=> prints 'Hi, Tom' to STDOUT.

Check out the Jekyll docs for more info on how to get the most out of Jekyll. File all bugs/feature requests at Jekyll’s GitHub repo. If you have questions, you can ask them on Jekyll Talk.

LIVE 2021 submission

Closing The Loop Early For Self-Improvable Graphical Systems

Author information

The dream: self-improvable systems based on structured representations

Building up from a human-friendly low level

The devil’s in the details

Minimal substrates for maximal progress?

Demo: camera panning lifted out of JavaScript

The basic instruction set

Demo: writing to the tree

Demo: conditional jump

And Now, The Shapes

Demo: camera panning internals

The story so far, and ahead

Food for thought

References

Welcome to Jekyll!