New Kind of Paper, Part Three
In previous parts (1, 2) I described New Kind of Paper, an app that will enhance the best medium for thinking – paper & pencil – with capabilities of a crazy advanced calculator.
In third part of the series, we will look at some prior art that serves as an inspiration for this line of research. Namely, we'll talk about Chalktalk, Magic Paper, APL, PyTorch and Pascaline.
Chalktalk
Chalktalk is a digital presentation and communication language invented by Ken Perlin. Its purpose is to serve as an aid for teaching various concepts. For example, when you want to talk about periodic motion, you can draw a pictogram of a pendulum that will immediately transform into a live swinging pendulum that can be interacted with in various ways. Here is a quick demo of Ken demoing it live:
Chalktalk is great example of a thought tool designed for live conversation about various topics. By accompanying your words with pre-made vocabulary of live visualizations, you can better explain concepts that would be difficult to convey in any other form. Don't take my word for it, just look at Ken demoing linear algebra for rotating an object in 3D:
These slighly-updated cave paintings, as Ken calls them, have to be understood in the context of Augmented Reality. Instead of drawing them on a digital blackboard, it is better to imagine them floating in the space around you as 3D objects. Interaction with them is done as naturally as possible – by using your hands, and of course your voice. Combining these objects together, as you have seen in the demo, you can get a wide range of behaviors that can easily remind you of a crude programming environment. If this sounds like futuristic sci-fi bullshit for you, I think you'll going to have a few eye-opening moments in the coming year or two. If you are interested in this kind of research, watch Ken's talk on How to Build a Holodeck.
Magic Paper
Michael Nielsen, the scientist, in a beautifully written article Magic Paper, demonstrates a Chalktalk clone in the context of artificial neural networks.
The demo is not that different from Chalktalk, but through implementing and using the new medium, Nielsen rediscovered why direct expression of thought has a strongly satisfying effect:
Instead of drawing a neuron, why not just hit “n”? Instead of drawing axes, why not just hit “g”, for “graph”? Wouldn't such keystrokes be faster and easier? This question has bothered me ever since first seeing Perlin demonstrate Chalktalk. Even as an audience member, I found it much more satisfying to see him draw a glyph than to just hit a key and have it appear. And this effect was even stronger as a user of Magic Paper.
This direct and intimate way of interacting with concepts feels much better than a blind manipulation of symbols on a piece of paper. But in the end, it is just another instance of visual programming paradigm that failed many times in past couple of decades. However, what has to be praised is that any live computational medium that enables you to explore your understanding of some dynamic system is a good medium, and therefore Magic Paper and Chalktalk are steps in the right direction.
So, if we throw out boxes-and-arrows, i.e. visual programming stuff, what's left? What is the essence of what we are trying do here? Is there a place for a more symbolic, but visually-enriched approach? Nielsen poses a profound question:
Mathematical ideas are conventionally expressed using notation and terminology developed using static media. Suppose, however, that mathematics had been invented after modern computers. This is perhaps difficult to imagine – after all, mathematics helped lead to computers – but let's do the thought experiment anyway. Might mathematical notation have developed in a different way? Would we instead have developed a dynamic, interactive notation more powerful than the static mathematical and linguistic notations in common use today?
APL
APL is one of the oldest programming languages that has been invented. You have probably seen a bite-sized implementation of the Game of Life in APL a hundred times by now. Even if you can't read it, you probably can appreciate the conciseness of the program. In comparison to the implementations of Life in many different programming languages you can found on Rosetta Code, APL is clearly the beast when it comes to expression of thought. This is not a coincidence, rather an intended design. Ken Iverson in his Turing Award Lecture Notation as a Tool of Thought presented APL as an unambiguous executable mathematical notation.
Similarly to another programming language dinosaur LISP, APL is still alive and kicking. There is now a whole family of languages modeled after APL, which is commonly referred to as an array-oriented programming paradigm, where array is a generalization of scalars, vectors and matrices. Because of the ongoing evolution, the the Game of Life in APL is now even shorter:
What strikes me as a suspicious coincidence is that these concise thoughts expressed in terms of linear algebra can be run extremely fast on current hardware accelerators such as GPUs, TPUs, Dojo or other so called neural processors. And since we can model a lot of things with arrays, these executable thoughts are very useful in many different domains.
PyTorch
Machine Learning can be viewed as a new kind of programming, or even more computing – Andrej Karpathy calls it, Software 2.0. The main selling point of Software 2.0 is the internal representation of the program, which can be optimized with math. Instead of hand-tuning variables of some generic algorithm, machine learning can do this all by itself via automatic differentiation. And since it is all linear algebra (with if-else & while-do !), it can be pretty damn fast and really useful. Think self-driving cars, smart AR glasses, landing rockets, self-sustaining bio-pods, endless synthetic art, and other magical shit.
PyTorch is one of many differentiable programming libraries used to built these new experiences. However, there are only few languages that has differentiability as a first-class language construct. Julia and Dex are forerunners, while Swift is slowly getting there. Libraries or languages, differentiability is here to stay and it will greatly extend our problem-solving abilities.
Pascaline
Pascaline, or Pascal's calculator is a mechanical calculator invented by Blaise Pascal in 1642 and perfected in following years. It could only do addition (and subtraction with a trick), but it was a marvel at that time. It was the first calculator to help with calculating taxes! Also science stuff, but Pascal's father was a tax man, so the operational design had to be simple enough to be used by non-technical people. This is how it was operated:
As you can see, it was operated by a stylus. This stylus was a pen with which the person was writing on a paper. Same hand-held interface for writing and computing – brilliant design. So brilliant, that some accountants expressed grave concern that they might be replaced by the technology! We might have perfected the computing technology in the past 400 years, but we are still the same bundle of nerves as ever.
Remix
If we combine everything together, what do we get? Is it simple to use as Pascaline? Does it feel magical like Chalktalk? Can we use it to go to Mars? Does it transform how we compute and think?
No.
Not yet. 😉
Continue to New Kind of Paper, Part Four…