New Kind of Paper, Part Five

In the previous parts (1, 2, 3, 4) written in 2021, I described New Kind of Paper – an app that enhances paper & pencil, the best medium for thinking, with the capabilities of a crazy advanced calculator.

In 2024, Apple introduced their spin on this topic under the name Math Notes. In this article, I will provide my deep praise (and even deeper hate) for their attempt to bring a bit of innovation into the UX of math.

---

Let's start with a simple example of how Math Notes works...

It *fucking* works! 🥹

At first glance, it may not seem like much, but in this brief example, a lot has happened. First, the handwriting recognition worked flawlessly, despite my awful handwriting. It perfectly recognized what I meant by those scratch marks. Second, after I wrote the trigger symbol, = (the equal sign), the expression was evaluated, and the result was inserted inline, mimicking my handwriting style. The inline insertion could be improved to better match the handwriting style, but overall, this is basically magic. With my writing magic wand, I have conjured up a little computational spell. 🪄

The fact that millions of people have this technology at their fingertips and pencil tips (works both on iPads & iPhones too), is a major miracle. The democratization of this technology is the hardest part, and it is in the hands of the people who know how to get shit done. Maybe not on the first try, but the third iteration... *chef's kiss*

Too Magical & Not Alive Enough

How do you know that the result is correct? The calculation is definitely correct, but how can you trust your calculator to recognize your badly handwritten "1" as "1" and not as "7"? You simply don't. This is essentially Apple claiming that their handwriting recognition is infallible. It's a rather bold claim, don't you think? I appreciate the confidence, but I don't believe we have reached that level of accuracy yet. Even basic calculators indicate which button you (mis-)pressed.

Why does it appear lifeless most of the time? If you only saw the first 90% of that demonstration, you wouldn't even know if this thing works. Honestly, it seemed dead most of the time. And when the result finally appeared, it was accompanied by flashy animation. 🫣 I love the animation, and I understand why it adds a nice touch to the initial version, but it's really just a distraction. This calculator should feel MORE alive!

Of course, talk is cheap, so let me demonstrate what I mean:

Not as polished, but it is alive! And snappy. And colorful. The feedback is immediate, and you can see the result as you write. This kind of feedback instills trust in the system and gives you a sense of control. Of course, Apple will fix these issues differently...

The first problem – whether the system recognized your handwriting correctly – can be solved with another iPadOS feature called Smart Script. It lets you beautify your handwriting – simply write an ugly "1" and it will progressively transform the scribble into neatly handwritten "1" or "7". The point is to inform the user, give them feedback on how the system recognized the symbol. This helps tremendously with the trust issue.

The second problem, the missing liveness, is even easier to solve – just ditch the "=" and evaluate the expression when the user pauses. Not too eager, not too lazy. This "interactive" mode can be simulated in Math Notes with this technique:

Scratch to Delete

*no comment*

The Math You Know™

2D notation

Math Notes supports 2D notation – you can write exponents as superscript, use fractions, etc. You know, the usual scary math stuff:

I know how crazy difficult this must have been to pull off. *Bravo!*

PEMDAS

PEMDAS – the order of operations: Parentheses, Exponents, Multiplication/Division, Addition/Subtraction. It's everywhere. It's in our textbooks, our calculators, programming languages and now even in Apple's Math Notes. *sigh*

The Future of Math Notation

Supporting the traditional handwritten math notation is no easy task. It is extremely messy, inconsistent and sometimes ambiguous. However, a calculator that wants to be useful must support ideas that have been forged for centuries. It is hard to change math.

When math started to become executable in form of software & hardware, we developed consistent notation that was shaped by keyboards, not by pencils. We migrated from 2D into 1D, misused some glyphs, and introduced constructs that made sense for computing. Math definitely adapted to this new medium.

Math Notes embodies even newer medium on which math can grow and change. While supporting existing traditional notation is a must, adding ability to define custom notation is an aspiration. We can't evolve math if the medium does not allow it. Today, the notation in Math Notes is fixed and it doesn't even cover a lot of useful math (e.g. calculus). This is fine in the short term, but if we are serious, we should start to think about user-definable 2D notation. Heck, even user-definable operators (e.g. ≈ or ⊙) would be a good first step.

While Math Notes isn't doing anything on this front yet, Apple leaned heavily into different direction...

Dynamic Scribbles

Since we are not stuck with static scribbles on the paper, Math Notes supports some dynamic behaviors, e.g. changing a numeric value just by dragging a slider.

Or graphing a function...

These are fantastic features that millions of people will love. With just a scribble and touch of a finger, I can solve non-trivial computational problems. Can you imagine the effort, if you wanted to achieve the same thing in e.g. Python? *Bleh..*

Graphs even respond to changing coefficients...

λλλ

One obvious omission in this part of Math Notes, is proper function definition. Define a function, see its graph, and be able to evaluate it with a specific input value. You know, something like this...

This notation for function definition (or lambdas/λ in comp-sci jargon) is pretty understandable and supports multiple arguments. Also, it opens up a route to primitive custom (infix) operators.

---

Solving Way Harder Stuff

All these features are hinting at a calculator that is a good companion for a mind that wants to solve problems that can be turned into a computation. So far, the capabilities of Math Notes are pretty limited – e.g. how would you sum up numbers from 1 to 100? That is a pretty easy computational problem, but you can't solve it easily with this type of calculator – yet.

However, "solving" is much broader topic than a straight-to-an-answer computation. What about problems that involve optimizing a function with thousand of parameters in an iterated manner? This problem-solving technique is currently limited to small set of smart people. Math Notes could unlock it for much broader, and younger audience. Like Excel did open up sophisticated computation for mere mortals. But this time in a much more humane way.

∞ Amount of Constructive Criticism...

I have been thinking about this type of calculator for many many years, and I am extremely happy to see this new kind of calculator in the wild. I want to see it used by everybody – from curious 5 year olds to PhD-level professionals in the science & engineering. Solving our personal problems and civilization-level ones too.

We can only see a short distance ahead, but we can see plenty there that needs to be done.

– Alan Turing

---

Do you have ideas about this kind of stuff?
Please share them online!

---