New Kind of Paper

Ideas are elusive, slippery things. Best to keep a pad of paper and a pencil at your bedside, so you can stab them during the night before they get away.

– Earl Nightingale

(Psst, there is also New Kind of Paper, Part Two, Part Three & Part Four…)

State of the Art Technology

Writing with pen is the first skill you learn in school. It is also one of the most important skills you learn in life, because writing enables you to record your thoughts and send them to another human. Your thought may affect a person that is thousands of kilometers away, and even hundreds years in the future. Recording thoughts is the most powerful technology our civilization ever created. And without writing, well, there wouldn't be any human civilization. Writing is here to stay, so we might look at how we can improve it.

Let's look at one specific mode of writing that people often do. You have a problem that can be expressed as an algebraic expression. For example you are a carpenter and you want to know volume of piece of wood that is 13cm by 13cm by 1m. Or you want to know price of 3kg of bananas if 1kg is for 2.44€. Or you want to know how much money you'll have in 10 years when you start with 1000€ and 3.5% annual growth. This is basic math that you can do with pen and paper without even using a calculator. However, calculator is equally useful tool as pen and paper because you can think faster by automating the boring parts. Okay, so let's look at some concrete example of this workflow:

You write down the expression and then you have to type it in the calculator. You'll probably do it by writing down partial results and typing them in the calculator to get the final result. This is a slow process that can be eased by using calculator with parentheses – you retype the expression char-by-char and you have your answer. Nowadays people don't really carry around physical calculators, but we have something more powerful – a computer. Computers have many forms – one can be a handheld calculator with touchscreen, which you could use in tandem with pen and paper to solve our math problem. Or you know, use pen and paper that are computers.

So let's write our problem again. This time with modern versions of our beloved tools.

We have black paper and white ink. Such innovation! But it looks identical as our first try. Even though we use computerized pen and paper, the new medium acts the same way as one that is 5000 years old. Thanks to machine learning, computers are now able to understand our handwriting, so we can enhance our pen and paper to be smart!

As you can see, this isn't great. The transcribed text has errors. We tried to write it in one shot and the handwriting looks ugly, so the recognition result wasn't as good. Maybe if we try to write it by chunks, we might get it right.

Entering text on computer is usually done with caret navigation – a blinking vertical line designates an insertion point – if you press a letter on a keyboard, it will be inserted at that position. With this mode we are mixing in different input system – keyboard – into our pen and paper land, but at least we can write what we wanted... However we lost the original form of our handwritten thought and we are left only with printed characters. We can do a bit better.

This was much better experience than the previous one. And we have both representations at the same time – original handwritten form, and also derived printed form that was generated after each stroke. Even though there was a lot of computation going on during the scribbling, we still don't know the answer of our math question. So let's try one more time.

Finally, this is it — we have everything we need. We have our original scribbles, which get translated into printed letters. On top of that, the computer recognized it is an algebraic formula and highlighted numbers and operators with different colors. Not only printed characters, but also individual strokes. And along the scribbling, we had formula evaluated the whole time. As soon as we finished last stroke on our problem statement, we already had the solution.

Pen, paper and calculator working seamlessly together to ease our thinking.

From Past To The Future

This new kind of paper understands what you are writing and try to be smart about what you want. So what can it do?

• it knows glyphs that are not on your keyboard, such as these: ÷ ∑ ⨂ ∴ → ← ‽ β
• it can spellcheck and beautify your handwriting
• you can add your own symbols (that can optionally expand to words)
• it knows about concept of variables – reuse values by attaching name to them
• it has unit conversion built in, so 3km ÷ 26min will be evaluated to 4.62km/h, and also you can ask for m/s
• it knows all functions (sin, cos, log, pow, etc.), all constants (π, ℏ, etc.), all useful formulas
• it can display graphics such as graphs, plots, diagrams, images or even interactive ball-and-stick model of a chemical compound
• it is programmable, i.e. you can write your own functions with recursion, conditionals and loops
• it can do differentiable programming and optimization
• it is a collaboration-first medium

In summary, it can do a lot and it is useful not only for simple math problems, but it has a real impact on every scientific field. However it has one fatal flaw – it doesn't exist yet. Various pieces of the tech exist, but they are not yet combined into this new transformative medium.

Computational capabilities that were mentioned in the feature list exist in Wolfram Mathematica. Mathematica is a phenomenal piece of software, there isn't a single doubt about that. However it has its surface form, i.e. the programming language, designed for a keyboard input method. Almost all languages for programming computers were designed for keyboards because keyboards become standard part of computers early on.

In the rich history of computer science there have been many interesting experiments that were stolen straight from the future. One of them was RAND Tablet. Here you can see a full demo of this marvel of technology.

It can display and edit charts and also source code of a program. And all without any keyboard attached to it! Of course there are a few things that are plainly wrong. For example, the surface for the pen and screen are separate. This was a downgrade from its predecessor, Sketchpad, which even today looks like an alien technology.

Sketchpad didn't have handwriting recognition. All textual input was done by a keyboard. A missed opportunity if you ask me. But Sketchpad started a revolution named Computer-aided design, which stood on strong graphical capabilities and direct manipulation of the content. Todays CAD systems have great software, but they are running on a cheap knockoff of pen and paper – desktop computer with a keyboard and a mouse.

RAND Tablet had handwriting recognition that was based on clever heuristic, i.e. no machine learning, and it required person to write slightly different than it would normally write with a pen and paper. As you have seen in the demo, the text editing was usuable, so it wasn't such a problem. However, if you paid close attention to the meaning of the text, you could see that it was an assembly language. And that was a problem. Assembly was designed for keyboard input, so editing it with a pen must have felt weird and unproductive.

This gets us to the question: Are there any programming languages that were designed for pen and paper? Yes, there was. A Programming Language, also known as APL. A language that started as a notation that was designed for human-to-human communication of computer programs, usualy written with pen on a paper, or chalk on a blackboard. To be fair, even APL suffered the transition from blackboard to keyboard. Original notation had sub-/superscripts and flow of the program was depicted with lines. When APL became a programming language, it was linearized, lost its flowchart-like visual, but kept its exotic glyphs. This was possible by using a special keyboard and printer that could print those characters. (source)

What set APL aside from other programming languages was its overal unified design. It was an array-oriented language – it had the array as a first-class construct. Operation Array.map that you may know from functional programming was implicit in APL. Adding 1 to an array of numbers was same as adding 1 to one number. This array-orientation and tersness of the notation ultimately led to a much higher-level operational semantics than other languages had in that era. Today, there is a family of APL languages, but the community is rather small and divided. Some APL ideas percolated down the history to the modern tensor/array-oriented differentiable programming, such as PyTorch, with which a computer can learn to recognize your handwriting.

What now?

It seems that pen and paper will finally get a deserved computational upgrade that was in the works for quite some time. And that is good because it is still the greatest medium to think with. But there are still a million questions to be asked on how to properly design it. Is there the one ultimate notation that can span from children to science gurus? Should the interactions and language of this new medium be standardized and taught in school? What would happen if children would use differentiable programming from the start? How can we ensure that the medium would be proactive in teaching the person? Would simple collaboration between its users be enough for it? I don't have these answers yet. But I will try to uncover them by making this new kind of paper a reality.

Continue to New Kind of Paper, Part Two…

---