Simon Zelazny's Blog

Built-in Japanese Dictionary App on iPhone

Mon, 06 Jan 2025 00:00:00 +0100

Reviving Dictionary.app on your iPhone (kind of)

Desktop Macs come with a superbly useful application called Dictionary.app, which is slowly getting shuffled out of sight in favor of ‘integrated’ search in Spotlight. I think this is a shame, since the quality of the dictionaries that come for free with Mac OS is simply outstanding.

I first got a Mac in Japan in 2005, while studying in Tokyo, and I really appreciated the fact that I could easily look up Japanese vocabulary on my computer, without internet access.

Unfortunately nowadays, there is no Dictionary app built-in to iOS. There is a ton of shovelware in the app store, and Apple’s nudging you to simpy use Spotlight to search for word definitions. I despise this modus operandi — I’d like to have a dictionary app where I go to look up words and that’s it.

Fortunately, Apple does provide a way to get an ersatz experience on your iPhone, courtesy of the Shortcuts app. I use it to look up unknown vocab or kanji on my phone when I’m reading a paper book, but it is in fact a universal, multilingual dictionary.

1. Open the `Shortcuts` application

You will have a bunch of pre-made junk here, but for now, just click on the + icon in the top-left corner to create a new shortcut. This will be our Dictionary application.

2. Select the “Scripting” tab

3. Select the “Ask for Input” action

4. Fill in the prompt to use when asking for a Text input

5. Click “Done”, then add another scripting step: “Set Variable”

6. Name the variable `word`, and have it set to “Ask for Input”

7. Click “Done”, then add another scripting step: “Show Definition”

8. Select the variable “Word” as the source of the definition. Click “Done” on top of the screen.

9. The shortcut is complete. When you click on in, you can input any word…

10. And see its definition in all the dictionaries available on your system

And that’s it! You can place the shortcut on your home screen stand-alone, so you have it handy whenever you need it. It works great with hand-drawn Japanese language input, so you can input kanji that you don’t know how to read.

My recreational programming in 2024: Mostly Standard ML

Sun, 15 Dec 2024 00:00:00 +0100

The year in review

To the extent that lack of free time permits, I try to do some recreational programming on the side, unrelated to my dayjob or any money-making concerns. I find it helps me retain some love for the subject matter, and keeps me sharp.

In the past year, I’ve gotten into Standard ML programming, using the excellent PolyML compiler, and got a bunch of tiny projects done. The approach I took was simple: just code. I consciously avoided getting into any kind of ‘industrial’ concerns, such as resuability, open-source contribution, etc. The idea was to simply produce a bunch of code and have fun doing it. Below is a list of fun things I built this year, with some reflections:

1. The `assert` testing library for PolyML

As I started out programming my little Standard ML projects, the first issue I ran into was the lack of quick-and-easy testing tooling. Yes, there is SMLUnit, but I find its approach too bureaucratic and ponderous for my taste. I wanted something that would let me 1) write a lot of tests easily without boilerplate; 2) provide readable output for tests results, without me having to define custom printing functions.

While the first requirement was subjective and easy to fulfill, the second one immediately posed problems, due to the fact that there is no standard interface to print arbitrary data in Standard ML. I find this to be an ironic situation, since most Standard ML environments provide a REPL, which does indeed know how to print out arbitrary data. Luckily, PolyML exposes its ad-hoc printing machinery as PolyML.makestring (although it must be called explicitly as PolyML.makestring – any kind of aliasing causes the magic to break), so I decided to let that fact guide me and focused my Standard ML programming on PolyML only. Ultimately, I think this was a good thing, as trying to make my code standards-compliant and compatible with all Standard ML implementations represented exactly the kind of ‘industrial’ busywork that I wanted to avoid.

About the library: there were a couple interesting things related to API design which did not strictly involve pretty-printing arbitrary data. First of all – what should be the type of a test? The simplest idea would be something like:

type testcase = (unit -> bool)

which would allow us to write tests of the form:

> val t1 = (fn () => 2 + 2 = 5) : testcase;

> t1 ();
val it = false: bool

This is sufficient to get going, but eventually one would like to see exactly which tests failed in a batch, and getting a list of results such as [true, true, true, true, false, true, false, true] is not great developer experience. So the next ‘easy’ thing would be to label our testcases so the runner could refer to them:

> type testcase = string * (unit -> bool)
> val t2 = ("two and two make five", fn () => 2 + 2 = 5): testcase

> (#2 t2)();
val it = false: bool

This is better, as now our hypothetical test runner can tell us which test failed. But it would still be nice to see the value which caused the test to fail. And so it’s at this point that we need to break out the pretty-printing. How would we design a testcase that prints out failed arguments? Maybe something like this?

> type testcase2 = string * (unit -> string option);
> val t4 = ("two and two make five", fn () =>
>    (if 2 + 2 = 5
>    then NONE
>    else SOME (Int.toString(2+2)))) : testcase2;

> (#2 t4)();
val it = SOME "4": string option

Okay, now we have a better informational API, but the test body itself has become incredibly unwieldy. We can try to extract the display function and only use it on a failing value … but that makes our test case typing much more involved. Something like this perhaps?

> type ''a testcase3 = string * (''a -> string) * (unit -> ''a option);
> fun runtestcase (desc, pp, testfun) =
>    case testfun () of NONE => (desc ^ " passed")
>                     | SOME v => (desc ^ " failed: " ^ (pp v));

> val t5 = ("two and two make five",
>           Int.toString,
>           fn() => if 2 + 2 = 5 then NONE else SOME (2+2));

> runtestcase t5;
val it = "two and two make five failed: 4": string

Okay, that reduced some of the machinery, but we still have to provide the failing value in the test body. It would be nice to extract this mechanism also. Maybe with an assertion function such as the following:

> fun assertEqual (got: ''a, expected: ''a, pp: ''a -> string) : string option =
>   if got = expected then NONE else SOME (pp got);

That lets us remove the ''a out of the testcase type and allows us to mix test-cases asserting on different types again:

> type testcase4 = string * (unit -> string option)
> val tests = [ ("int addition", fn () => assertEqual(1+1, 3, Int.toString)),
                ("concatenation", fn () => assertEqual("a"^"b", "abc", fn x => x)) ];

> map (fn (desc,t) => (desc, t ())) tests;
val it = [("int addition", SOME "2"), ("concatenation", SOME "ab")]:
   (string * string option) list

This is nice, but for maximum usability, we’d like the test framework to do all our pretty-printing for us, since there is no built-in pretty printer even for a list of ints, much less for any user-defined datatypes. Also, it would be good to test for non-equality and for exceptions thrown, and to have all the data that we assert on printed out in the test log.

To that extent, I ended up with a test structure which exposes test case constructor functions as well as test assertions, and leverages the type system to give you confidence that a test case must contain an assertion and cannot silently pass.

So, for example, here is a list of tests asserting that Base64 decoding errors are thrown by the Base64 library under test:

val errorTests = [
  It "bails out when input is ascii but not from the base64 set" (
    fn _=> (B.DecodeError "Input not Base64") != (fn () => B.decode "!@#%")
  ),
  It "bails out when input is too short" (
    fn _=> (B.DecodeError "Invalid/incomplete sequence") != (fn () => B.decode "a")
  ),
  It "bails out on non-ascii input" (
    fn _=> (B.DecodeError "Input not ASCII") != (fn () => B.decode "Żółw")
  )
]

Note the use of the != assertion, which takes an Exception on the left and a thunk on the right, and asserts that the thunk throws the exception when evaluated.

Here are some tests of the example text from the Base64 wikipedia entry:

val exampleDecodingTests = [
  It "decodes the wikipedia example: Man" (
    fn _=> B.decode "TWFu" == Byte.stringToBytes "Man"
  ),
  It "decodes the wikipedia example: Ma" (
    fn _=> B.decode "TWE=" == Byte.stringToBytes "Ma"
  ),
  It "decodes the wikipedia example: M" (
    fn _=> B.decode "TQ==" == Byte.stringToBytes "M"
  )
];

The provided test runner lets us run all the tests we defined like so:

fun main () = runTests ( exampleEncodingTests
                       @ paddingTests
                       @ exampleDecodingTests
                       @ errorTests
                       @ roundTripTests)

If we modify one of our tests to fail

  It "decodes the wikipedia example: Ma" (
    fn _=> B.decode "TWE=" == Byte.stringToBytes "Maxx"
  ),

and run the test suite with poly, we get:

FAILED decodes the wikipedia example: Ma
	fromList[0wx4D, 0wx61] <> fromList[0wx4D, 0wx61, 0wx78, 0wx78]

TESTS FAILED: 1/17

make: *** [Makefile:6: test] Error 1

I’m quite happy with this design and I consider it the only one that’s “production ready” from among this year’s recreational projects. It’s a one-file library, and including it in your code is as easy as opening Assert. I used it in all my subsequent Standard ML codebases.

2. A Base64 encoder/decoder

I took on this project to answer the question of just how much speed one can expect of Standard ML with real-world tasks, such as text encoding.

It turns out that naively implementing the logic leads to very poor performance, primarily caused by massive memory churn on allocations and deallocations of intermediate datastructures. I took a look at various other implementations and found that the SML/NJ algorithm was blazing fast.

Here is my benchmark with Poly/ML used to compile and run the algorithms:

/bin/time ./bench b  # My algorithm
8.17user 1.82system 0:08.06elapsed 124%CPU (0avgtext+0avgdata 210612maxresident)k
0inputs+0outputs (0major+198647minor)pagefaults 0swaps

/bin/time ./bench s  # sml/nj algorithm
0.88user 0.38system 0:01.28elapsed 99%CPU (0avgtext+0avgdata 20936maxresident)k
0inputs+0outputs (0major+43224minor)pagefaults 0swaps

And here is the same benchmark run under mlton:

/bin/time ./bench b    # my algorithm
1.88user 1.60system 0:03.52elapsed 99%CPU (0avgtext+0avgdata 535832maxresident)k
0inputs+0outputs (0major+180440minor)pagefaults 0swaps

/bin/time ./bench s     # sml/nj algorithm

0.34user 0.12system 0:00.47elapsed 98%CPU (0avgtext+0avgdata 47344maxresident)k
0inputs+0outputs (0major+13291minor)pagefaults 0swaps

Some interesting things to note: 1) simply using mlton to compile your code gives you a 4x speed improvement for the exact same task; 2) Using mlton to produce the ‘release’ version of your code, while leveraging polyml for quick compilation & test-turnaround is exactly the kind of ‘industrial’ activity I wanted to avoid, but the polybuild guy has put together some nifty harnesses to make it doable; 3) ultimately, any string-mangling you end up doing in Standard ML is going to get smoked by python (because it calls out to c):

/bin/time python3 ./test/bench.py
0.09user 0.02system 0:00.12elapsed 99%CPU (0avgtext+0avgdata 15484maxresident)k
0inputs+0outputs (0major+5731minor)pagefaults 0swaps

and smoked all the worse by the gnu coreutils:

/bin/time base64 ./bytes > /dev/null
0.00user 0.00system 0:00.00elapsed 25%CPU (0avgtext+0avgdata 1920maxresident)k
96inputs+0outputs (2major+108minor)pagefaults 0swaps

So the outcome of this experiment was not super encouraging – even the best-written algorithm, compiled with a powerful sci-fi compiler, still loses out to the python stdlib. On the other hand, I had a fun time testing and implementing a well-known standard, so I got my kicks nonetheless.

3. Sqlite3 FFI binding for PolyML

This project was very educational, rewarding, and just plain fun. I actually started working on it on my wife’s computer during our family trip to Japan. With only emacs, the sqlite3 documentation, and the PolyML source and docs, I was able to put together a basic but functional sqlite3 library, complete with a reliable test suite.

I don’t have much commentary here. I somehow plodded through the various components based on trial-and-error, trying to figure out how the outdated PolyML FFI tutorial relates to the current FFI implementation. Ultimately, I was able to work things out based on the PolyML source code.

Here are a couple of the tests I eneded up writing:

  It "can read a row (unicode literal)" (
    fn _=>
       let val db = givenTable "create table f (a int, b double, c text)";
           val _ = S.runQuery "insert into f values (?,?,?)" [
                 S.SqlInt 1,
                 S.SqlDouble 2.0,
                 S.SqlText "こんにちは、世界"] db;
           val res = S.runQuery "select * from f" [] db;
           val _ = 1 =?= length res
       in case hd(res) of
              [S.SqlInt 1,
               S.SqlDouble _,
               S.SqlText "こんにちは、世界"] => succeed "selected"
            | other => fail ("failed:" ^ (PolyML.makestring other))
       end),


  It "returns proper codes on constraint violation" (
    fn _=>
       let val db = givenTable "create table f (a int)"
           val r0 = S.SQLITE_OK =?= (S.execute "create unique index fi on f (a)" db)
           val r1 = S.SQLITE_OK =?= (S.execute "insert into f values (1)" db)
           val r2 = S.execute "insert into f values (1)" db
       in r2 == S.SQLITE_CONSTRAINT
       end)

4. Serpent Encryption in Standard ML

“Where is this all going?”, you might ask. All of the above components actually were supposed to build up to a working client of a certain p2p chat protocol – the one that posits using Serpent, the runner up to the AES spec.

This project, like the sqlite3 library, had me streching myself to work in a domain which I’m not super familiar with. (don’t roll your own crypto, they said). Aside from whatever difficulty was inherent in the problem domain itself, I had to do a bit of online archaeology to ensure I’m implementing the right thing. A good test library (as implemented by me) helped a lot with making sure I wasn’t regressing, but the bulk of the work consisted in finding working implementations of Serpent and ensuring mine was compatible.

This was not as easy as it sounds, as Serpent is mostly forgotten, but there are several implementations floating around: the original NIST implementation and its variants, some educational variants, and some free-floating c libraries. Endianness problems abound with the NIST data going one way, regular usage going another. Some implementations do CBC, others don’t. Etc, etc.

Long story short, I was able to get an implementation working, which, despite having a messy interface, is quite performant and well-tested. I sure as hell wouldn’t encrypt anthing sensitive with it, but it meets the spec. It does CBC.

I had a lot of fun implementing the Osvik S-boxes, even going as far as to recreate the table layout in the sml code:


fun sBox0 (blockWords : block) : block =
    let
        val (r0,r1,r2,r3) = mkRefs blockWords;
        val r4 : word32 ref = ref 0w0
    in
       (r3 ^= r0; r4 := !r1;
        r1 &= r3; r4 ^= r2;
        r1 ^= r0; r0 |= r3;
        r0 ^= r4; r4 ^= r3;
        r3 ^= r2; r2 |= r1;
        r2 ^= r4; r4 =~ r4;
        r4 |= r1; r1 ^= r3;
        r1 ^= r4; r3 |= r0;
        r1 ^= r3; r4 ^= r3;


        (!r1, !r4, !r2, !r0))
    end

I think this is one of those instances where Standard ML’s facility with defining new operators proved to be a big benefit. After defining the mutating operators ^=, &=, |=, and =~, I was able to follow along with the paper and mechanically input the paper’s original syntax into my code.

5. Roguelike etudes

This is not a notable project by any means, but it was just me trying to put together a bunch of little toy programs without getting bogged down in detail. Despite actually writing a playable roguelike being a dream of mine since high school, I didn’t achieve much in this regard.

I did spend some time programming the toys with my kids. They provided me the impetus for adding a bit of visual flair, taking my version of 2048 from this:

To this:

Although nothing resembling a rogulelike emerged from this project, it provided fun for me and the kids. Standard ML turned out to be a nice lanague for this type of toy.

6. Working through “ML for the Working Programmer”

I did a bunch of exercises from the first half of this book. They are quite hard, but rewarding. To aid my understanding, I ended up writing little utilities to visualize the data structures involved.

Standard ML actually makes it quite easy to interact with POSIX-world, and with a few functions such as:

type cmdBuilder = (string -> string list);

fun page tmpfile =
    ["(dot -Tpng ", tmpfile, " | 9 page -R)"];

fun save tmpfile =
    ["dot -v -Tpng -ooutput.png ", tmpfile ];

fun drawTree (fmt : 'a -> string) (cmd : cmdBuilder) (t : 'a tree)
    : Unix.exit_status =

I was able to visualize all the structures that were being manipulated in the section on trees and functional arrays. That allowed me to take this tree:

> tree4;
val it =
   Br
    (4, Br (2, Br (1, Lf, Lf), Br (3, Lf, Lf)),
     Br (5, Br (6, Lf, Lf), Br (7, Lf, Lf))): int tree

> drawTree Int.toString page tree4;

And get this graphic:

This really helped when the structures got large and complicated. In general, working on this book made me appreciate how, when learning new topics, it helps to ‘see’ them from multiple perspectives. Being able to see the subject matter in graphical form made it interesting even for my kids.

My interest in Standard ML slowly petered out as I was working on this book. I realized that trying to get a ‘real’ project off the ground will require the kind of ‘industrial’ thinking that I’d been avoiding throughout. It would mean integrating various 3rd-party libraries and getting them to build together, deciding on module dependency structures, building smlnjlib on polyml, etc. That kind of stuff not qualifying easily as “recreational programming” in my book, I decided to try something else next.

7. Classic Computer Science Problems in Python

Since I use Python at work, I figured it wouldn’t hurt to do some recreational coding in it. I picked this book based on Amazon suggestions and Henrik Warnes’s review. I’ve completed six out of nine chapters and feel I’ve gotten a good refresher on classic CS, along with some new Python tricks up my sleeve.

Like Henrik, while I have some misgivings about this or that aspect of the book, I recommend it to colleagues as a high-density refresher on the kinds of things you probably don’t work on at work.

8. Other loose ends

Several friends have asked me whether I’m doing Advent of Code this year. The short answer is: I love the idea and implementation of Advent of Code, but the timing is so cruel as to be sadistic. In the run-up to Christmas, it’s really hard to get an hour a week to devote to recreation programming, much less an hour a day.

Good luck to all of you who are participating this year!

Mechwarrior 2 on Dosbox: Fixing the “Press any key to exit“ problem

Wed, 20 Nov 2024 00:00:00 +0100

If you’re trying to play Mechwarrior 2 for Dos under dosbox and find that the game hangs after a mission ends, displaying the message “Press any key to exit” on top of the screen, this is the solution:

Grab dosbox-x, compile it, and enjoy your Mechwarrior 2.

Working grayscale color filter on linux (Xwindows)

Mon, 20 May 2024 00:00:00 +0200

Picom compositor to the resuce

I’m a big fan of grayscale color filters on computer screens. I have them on on my iPhone and on my work laptop (Mac OS). However, I’ve never managed to turn my linux desktop black-and-white, in spite of periodic attempts every couple of months.

To be precise, no xorg.conf setting worked reasonably for me. If I changed the Depth or DefaultDepth setting in my Screen section, I’d get a ‘grayscale’ experience, except that most applications would render images or text as big black squares. I know that the KDE compositor KWin has grayscale filter functionality, but I’ve never been super keen on pulling all those QT dependencies into my barebones WM setup (openbox + kupfer + xfce4-panel).

The solution turned out to be standalone compositor picom, used in conjuction with a user-defined shader.

First, define the shader as follows:

#version 330

in vec2 texcoord;
uniform sampler2D tex;
uniform float opacity;

vec4 default_post_processing(vec4 c);

vec4 window_shader() {
	vec2 texsize = textureSize(tex, 0);
	vec4 color = texture2D(tex, texcoord / texsize, 0);

	color = vec4(vec3(0.2126 * color.r + 0.7152 * color.g + 0.0722 * color.b) * opacity, color.a * opacity);

	return default_post_processing(color);
}

Then, in your xorg startup script (such as .xinitrc), start picom and point it at the above definition. So if you saved it under $HOME/src/picom-grayscale-shader.glsl, that’s what you pass in:

picom --backend=glx --window-shader-fg=$HOME/src/picom-grayscale-shader.glsl &

And presto! A fully usable grayscale desktop.

The source for the fix can still be found in one of the github issues for the project, but I’m noting this down here because the reference is hard to find.

Unicode string literals in PolyML

Sun, 28 Apr 2024 00:00:00 +0200

A small hack with a large ergonomic payoff

As I wrote previously, I’ve put away the SML# and got back to recreational programming in plain old Standard ML. I really love the spartan feeling of the language and the fact that it doesn’t allow for too much fanciness (module-level magic aside), while still enabling a nice, high-level functional-programming experience.

One “spartan” aspect of Standard ML which does not constitute a nice experience is its insistence on all string literals containing only ASCII characters. This is limiting and frustrating, especially for those of us who use non-English alphabets daily. Now, thanks to the clever design of UTF-8, unicode text encoded as UTF-8 will get displayed nicely on your screen as a Standard ML string.

Here is a demo:

$ cat emoji.txt
🙈🙉🙊


$ poly
Poly/ML 5.7.1 Release

> val textFile = TextIO.openIn "emoji.txt";
val textFile = ?: TextIO.instream
> val (SOME s) = TextIO.inputLine textFile;
val s = "\240\159\153\136\240\159\153\137\240\159\153\138\n": string
> print s;
🙈🙉🙊
val it = (): unit

As you can see, if we can get the unicode text into a string, we can display it thanks to the pervasive unicode support in our OS. That’s good enough for me. However, it’s not easy to get that unicode text into a string, outside of reading streams from a file.

val monkeys = "🙈🙉🙊";
poly: : error: unprintable character \240 found in string
poly: : error: unprintable character \159 found in string
poly: : error: unprintable character \153 found in string
poly: : error: unprintable character \136 found in string
poly: : error: unprintable character \240 found in string
poly: : error: unprintable character \159 found in string
poly: : error: unprintable character \153 found in string
poly: : error: unprintable character \137 found in string
poly: : error: unprintable character \240 found in string
poly: : error: unprintable character \159 found in string
poly: : error: unprintable character \153 found in string
poly: : error: unprintable character \138 found in string
Static Errors

I’ve spent some time digging into the PolyML code to make the above possible. With my patch for unicode string literals applied, the above interaction is legal syntactically!

$ poly
Poly/ML 5.9.1 Release (Git version v5.9.1-64-ga71e81c1)
> val monkeys = "🙈🙉🙊";
val monkeys = "🙈🙉🙊": string
> String.size monkeys;
val it = 12: int

As you can see from the result of the call to String.size, my patch does not magically make PolyML strings unicode-aware, but the ergonomic improvement is fantastic nevertheless.

The .patch file linked above also contains tests, but if you’re only interested in the implementation code, here is the relevant diff:

diff --git a/basis/String.sml b/basis/String.sml
index a2b2a7ab..9bca902a 100644
--- a/basis/String.sml
+++ b/basis/String.sml
@@ -158,7 +158,7 @@ local
         fun isHexDigit c =
             isDigit c orelse (#"a" <= c andalso c <= #"f")
                  orelse (#"A" <= c andalso c <= #"F")
-        fun isGraph c = #"!" <= c andalso c <= #"~"
+        fun isGraph c = #"!" <= c andalso c <= chr 255
         fun isPrint c = isGraph c orelse c = #" "
         fun isPunct c = isGraph c andalso not (isAlphaNum c)
         (* NOTE: The web page includes 0 <= ord c but all chars satisfy that. *)
@@ -719,7 +719,7 @@ local
         case getc str of (* Read the first character. *)
             NONE => SOME("", str) (* Just end-of-stream. *)
           | SOME(ch, str') =>
-                if ch < chr 32 orelse chr 126 < ch
+                if ch < chr 32 orelse chr 255 < ch
                 then NONE (* Non-printable character. *)
                 else if ch = #"\\"
                 then (* escape *)

Wrapping up `Practical SML#'

Sat, 24 Feb 2024 00:00:00 +0100

Review of chapters 9-11 & overall thoughts

For Context: All the previous posts in this series

Looking back

In July 2023, I’d just received my copy of “Practical Programming with SML#” (SML＃で始める実践MLプログラミング), and decided to blog about each chapter as I read it, as a form of making SML# a bit more accessible to the English-speaking blogosphere.

I started out with a pretty good cadence, working through the first three chapters before the end of July. Then, my pace flagged, and chapters 4-8 took me the rest of the year to write up.

It’s now February 2024 and I’m wrapping up this series, without having reviewed the most meaty and intriguing final three chapters. The truth is that my enthusiasm towards SML# as a hobby has gone down considerably, and most of my (very scarce) recreational programming time is going towards plain SML these days.

The rest of this post will contain a brief synopsis of the last three chapters, and then my overall impressions of the SML# ecosystem, based on my admittedly cursory engagement.

Chapter 9: Cooperating with Databases

This chapter walks us through the process of defining and persisting our application data in a relational database. As the other chapters in the book, it’s very complete, and has us do everything from establishing a database connection, through to INSERT and UPDATE commands, handling conversion and query results, and finally running an analysis of the COVID data (imported using the JSON conversion features introduced in the previous chapter).

Having felt enough professional pain from tight coupling of application models and logic to the DB layer (ActiveRecord, SQLAlchemy, etc., and to a lesser extent Ecto), I was not exactly super enthused about this chapter and the approach taken by the language designers.

Essentially, as I understand it, SML# uses its built-in reflection and dynamic typing capabilities to build SQL statements based on SML#-level type information. This means that the entities on the SML# side and the entities on the SQL side must align 1:1 in terms of naming and semantics. Perhaps this approach is in line with the KISS philosophy, and forces application developers to keep their application-side code wholly reflective of the reality in the DB.

Maybe I’m spoiled from having been exposed to the Haskell way of solving these issues with Typeclasses, but to me both the JSON conversions and the SQL mappings are a kind of false economy, making the easy things easier but the hard things impossible. I’d prefer to do a bit more work up-front defining converters or parsers (as the Elm language forces you to), than to acquiesce to 1-to-1 mappings, and therefore dependencies, between my application code and serialized data.

(To be pedantic: it’s possible, of course, to have a layer of SML# code that serves as the ‘parser’ layer, and then create truly pure models from this ‘parser’ layer, but practically speaking no one is going to go to these lengths in the name of elegance or flexibility.)

This is the chapter that probably “lost me” to the cause. But, being so close to completing the book, I decided to skip implementing the code in this chapter and jump straight to parallel programming.

Chapter 10: Parallel Programming

Having used Haskell for several years, and then Erlang and Elixir for most of my programming career, I came to this chapter with high expectations. Unfortunately, I wasn’t really able to get my hands dirty with the material, because Pthread.Thread.join would keep hanging my smlsharp session, and only a SIGKILL could get it unstuck. Yes, the thread I was trying to join had finished doing the work. Yes it did print the result after getting SIGKILLed! (You might be thinking that I should have gone and debugged this strange and interesting concurrency bug. That’s true, but I really felt at this point that the juice was not worth the squeeze. Keep reading for more on this topic).

I really wanted to see the pretty ray-traced pictures, so I implemented the single-threaded raytracer in regular SML, and compiled with polyml. It compiled and ran like a charm!

At this point I was already disheartened enough that I didn’t proceed with the MassiveThreads parallel raytracer discussed in the rest of the chapter.

But to summarize the remaining part: in contrast with OS-level threads that regular SML implementations use, SML# offers drop-in support for a “green-thread” system called MassiveThreads. These are much cheaper to initialize and run, enabling finer-grained parallelism and better utilization of CPU resources.

Chapter 12: Techniques of Developing Practical Systems

This chapter is a synthesis of all the previous ones: we have a fully-fledged C-integration with Cairo (producing PDFs no less), a database in sqlite, and command-line parsing. The chapter sets out to prove that we can realistically apply SML# to the real-world task of plotting datapoints from a relational database.

Epilogue

The epilogue drives home the key points made by the authors in the course of the book. There are four fundamentals of SML:

1) Think in types
2) Write functions while keeping recursive structures in mind
3) Express the problem with data definitions
4) Use make and incremental compilation

And four special characteristics offered by SML#:

5) Access software libraries via C
6) Ingest external data safely with dynamic typing
7) Directly program the database
8) Use parallel computation features, harnessing multicore CPUs

My thoughts on the book, the language, and the ecosystem

First of all, a disclaimer: I read the book solely for personal enjoyment, in a hobbyist capacity. Perhaps working through it in either an academic or a professional setting, with experts on hand to talk to, would have been a different experience. I also didn’t implement any larger program apart from the examples and exercises from the book.

With that perspective clear, here are my thoughts.

1. This was one of the best programming books I have read

I would place it right alongside “Common Lisp: A Gentle Introduction to Symbolic Computation”, “Structure and Interpretation of Computer Programs”, and “Erlang Programming”, my three personal favorites.

It’s not just a well-written book, but it’s a book that accomplishes what it sets out to do: impart the authors’ knowledge on the reader. There’s never any doubt as to what the authors are trying to convey, and every bit of code has an explanation. My Japanese reading skills are nowhere near native level, yet I never felt lost or confused by grammar or vocabulary choices.

There is a lot of code in the book, interspersed with discussion, so the pace is fast and always feels engaging. The SQL chapter is a bit of an odd one out, sometimes coming stylistically closer to a reference manual than to a tutorial.

Then there are the exercises, which directly reinforce and often expand on the material from the preceding chapter. In my mind, this is the gold standard for a practical programming book. I really enjoyed doing these exercises, and thanks to this I retained much more of the presented information.

2. The language is an unqualified ergonomic improvement over Standard ML, with some raw areas

SML# the language is the reason I bought the book, and I wasn’t let down. It improves on the standard in many subtle but very pleasant ways, from thorough Unicode support to the ability to selectively match on records, really bringing the developer experience into the 21st century when it comes to programming in the small.

The Dynamic, SQL and Foreign extensions to the language feel a bit raw, like the authors are sharing with us the internal implementation of features not yet fully completed. These modules are very powerful but somewhat arbitrary, like Go’s special treatment of maps and slices, or StandardML’s own special “equality types” and math operator overloading. It feels like some parts of the system have been given superpowers, but the user has only the ability to plug in to these superpowers, and not to extend them.

I can imagine these features seeing more involved development. On one hand: allowing user-level extensions (such as custom Dynamic.fromXYZ converters), and on the other, building higher-level features (such as deriving-style code generators) that use these extensions under the hood. We already know it’s possible, given how smoothly the JSON and SQL intergrations work.

3. The tooling is underwhelming given its ambitions

There is a lot to like in the smlsharp tooling. The system compiles cleanly (*once you sort out massivethreads), includes high-quality SML libraries (smlnj-lib, smlunit) and tools (smlyacc, smllex, smlformat), and the make integration is a very welcome change from endless language-specific build tools. And yet, there are a couple things that prevent the experience from really taking off.

First: the compiler is just slow. The authors at several points make the case that the main goal for SML# has always been extending the language in a practical direction for usability, and this came at the cost of performance work. I appreciate this approach and think it’s the correct one. This approach gave us Lisp and Erlang and Haskell, and each of these languages is —nowadays— sufficiently performant for real-world usage.

Still, with that understanding, the compilation process is almost unacceptably slow for day-to-day use. Here are the compilation times for the ray-tracer program:

% time mlton main.sml
Warning: main.sml 98.14-98.21.
  Declaration is not exhaustive.
    missing pattern: NONE
    in: val SOME y = Int.fromString (hd args)

real	0m4.188s
user	0m2.459s
sys	0m1.482s
% time smlsharp main.sml
main.sml:98.14(2768)-98.46(2800) Warning: binding not exhaustive
      SOME y => ...

real	0m1.069s
user	0m0.868s
sys	0m0.195s
% time polyc main.sml
main.sml:98: warning: Pattern is not exhaustive. Found near val (SOME y) = Int.fromString (hd args)

real	0m0.156s
user	0m0.112s
sys	0m0.045s

So… not as slow as mlton, but still several times slower than polyc. This really adds up, and works against the ‘practical’ bent of the language. Fast compilation, and therefore developer feedback, is why Turbo Pascal was so popular back in the day :)

Second: The make-based incremental build system has some warts, and the .smi interface-file scheme feels like a throwback to the early ’90s. I don’t enjoy having to write practically the same code in two places, and that’s effectively what you end up doing. Again, maybe I’m spoiled by Haskell’s module system with explicit imports and exports, but the .smi-file dance feels to me like something the compiler/build-system should be doing for me.

Also, regarding the clunkiness of the make system, here’s a line from the last chapter of the book: the chapter which, mind you, demonstrates the full extent of real-world programming with SML#:

$ smlsharp -MMm dbplot.smi > Makefile
$ sed -i.orig -e '/^LIBS/s/$/ -lcairo/' Makefile
$ make

First, we make some edits to our interface file, dbplot.smi. As I noted above, this seems like an unnecessary step, at least for most usage. (I know there is a special provision for mutliple files exporting the same interface). Anyway, okay, we edited the .smi file, so we have to regenerate our Makefile. Okay, let’s say we agree to this step. But the next step is unacceptable: we have to go and re-write our autogenerated Makefile with sed.

This is a violation of DRY: I don’t want to have to re-run sed on my makefiles everytime I change an interface file. I want to specify somewhere that my code depends on -lcairo and not have to continuously re-jigger the build artifacts by hand. I even made a Pull Request with a proposal for a fix… which brings us to the worst aspect of smlsharp, and the real dealbraker when it comes to real-world adoption. The lack of publicly active users.

4. The community is inactive, to put it politely

For a project of such high quality and high visiblity, SML# feels like a ghost town. Since the book came out in 2021, there have been a meager 23 commits to the github master branch, the last one in March 2023. There isn’t any discussion on the PRs that folks (including myself) have put up, and the github forums in both Japanese and English have had zero activity since early 2022.

To compile massivethreads on a reasonably modern GLIBC, you need to find the debian patch in a developer’s personal fork. This hasn’t been addressed in the smlsharp master branch README.

There was some activity on Japanese programming Twitter around the time the book came out, but sadly no one really ran with it for a longer period of time.

In general, from my 7-month long engagement with smlsharp, I got a taste of that famous Japanese feeling of wabi: the presense of something brilliant, deep, meaningful, and yet always distant, empty, and already fading away.

So there you have it. For my recreational programming, I’m going to be sticking with the “standard” Standard ML and the blazing fast poly compiler. If you’re up for some SuccessorML-style experimentation, you’d do much better to look at LunarML, which is very active and very, very promising. And also Made in Japan!

Working Review of "Practical ML Programming with SML#" (Ohori, Ueno), CHAPTER 8

Sun, 31 Dec 2023 00:00:00 +0100

Accessing External Data

Before the year ends, let’s take a look at Chapter 8, titled “Accessing External Data”. This chapter teaches us how to read and interpret files on disk, as well as how to parse JSON data. Additionally, we are introduced to ML-style error handling, via explicit handle expressions and pattern-matching on exn constructors.

File I/O

This section gives us a very quick run-down of the TextIO structure, a Standard ML Basis module. We implement a very simple file-copy procedure.

Handling errors using the Exception mechanism

This section is quite long, but it contains a lot of information about Standard ML’s exception mechanism. For those of you who don’t know what it looks like: SML uses the same single datatype for all exceptions, and this datatype can be extended by the user via exception declarations, such as the following:

exception FailedToDownload of string;

Now, we can raise instances of these exceptions anywhere in our code, and they will stop the flow of control and bubble up to the nearest handle expression in the call-stack:

(raise FailedToDownload "howto.txt")
handle
  (FailedToDownload filename) => print ("Failed to download: " ^ filename ^ "\n");

This is a nice way to do non-local control flow, such as early-returning from a map or fold once the result is known. This meshes nicely with an imperative, effectful style of programming:

# fun find (predicate: 'a -> bool) (list: 'a list): 'a option =
>  let
>    exception Found of 'a
>  in
>   (app (fn el => if predicate el then raise Found el else ()) list; NONE)
>   handle Found x => SOME x
> end;
val find = fn : ['a. ('a -> bool) -> 'a list -> 'a option]

# find (fn x => x > 10) [1,2,3,4,5,6,7,8,9,10,11,12];
val it = SOME 11 : int option

# find (fn x => x > 10) [1,2,3];
val it = NONE : int option

It is also the preferred way of signaling exceptional circumstances, such as I/O failures. For example, trying to open an non-existent file raises an IO.Io exception, with some attached information about the fault.

# TextIO.openIn "missing.txt"; ();
uncaught exception IO.Io: openIn at src/smlnj/Basis/IO/text-io.sml:807.24(25074)

# (TextIO.openIn "missing.txt"; ())
> handle
>  (IO.Io{name, function, cause}) => print (concat ["IO error ",
>                                                    name, " ",
>                                                    function, "\n"]);
IO error missing.txt openIn

To round out the intro to SML-style error handling, the authors go on to demonstrate how to implement a generalized IO-error handler. This handler is then extended with ‘finalizer’ functions, which play the role of a (syntactically unavailable) finally clause.

Overall, the first part of this chapter is a rehash of exceptions and error handling in Standard ML. The next part is a radical departure from the standard.

Reading JSON Data

This section introduces how SML# implements dynamic typing at runtime. Yes! Even though SML# is compatible with Standard ML, it is capable of runtime typing, via the built-in Dynamic module, which is a very interesting extension to the language.

The thing to note, is (as far as I’m aware), the scope of dynamic typing is restricted to:

existing types
views into record types (i.e. particular fields)
JSON (this chapter)
SQL

That is to say: on one hand we can’t get in on the “magic”, as it is contained within the language runtime, and implement our own Dynamic converters, say, from a binary format like Protobufs. On the other hand, we can be sure that no 3rd-party SML# code will spring some outrageous dynamic typing scheme on us. To me, this is a reasonable point in the design space, although it definitely whets the apetite for what could be done if Dynamic was more open to user-level tweaks.

Here’s how the dynamic typing works, in a nutshell. Firstly, we can dynamically recover the types of existing values:

# open Dynamic
# val one_int = dynamic 1;
val one_int = _ : void dyn
# val one_real = dynamic 1.0;
val one_real = _ : void dyn

The type returned by dynamic is void dyn, which means it’s a dynamic value without a particular type-level interpretation attached. We have to provide the interpretation, using the built-in function _dynamic EXP as τ:

# _dynamic one_int as int;
val it = 1 : int
# _dynamic one_real as real;
val it = 1.0 : real

However, type coersion is not possible, and, just like all failed _dynamic invocations, raises a RuntimeTypeError.

# _dynamic one_real as int;
uncaught exception PartialDynamic.RuntimeTypeError at (interactive):7.0

Next up, we have views into record types:

# val a_car = dynamic { make = "Ford", model = "T"};
val a_car = _ : void dyn
# val a_make = _dynamic a_car as {make: string} dyn;
val a_make = _ : {make: string} dyn
# view a_make;
val it = {make = "Ford"} : {make: string}

As you can see, we have dynamically constrained the record type to just the fields that interest us, and subsequently materialized the data with view.

And finally, let’s take a look at ‘parsing’ JSON, using example data to inform the runtime type assigment.

# val json_car = "{\"make\":\"Ford\",\"model\":\"T\"}";
val json_car = "{\"make\":\"Ford\",\"model\":\"T\"}" : string
# val dyn_car = Dynamic.fromJson json_car;
val dyn_car = _ : void dyn
# val dyn_model = _dynamic dyn_car as {model: string} dyn;
val dyn_model = _ : {model: string} dyn
# view model;
val it = {model = "T"} : {model: string}

This mode of operation means that we can’t use SML#’s runtime typing as a silver bullet for ingesting external JSON data. We have to actually provide the runtime with an example of what we’re trying to extract, which means that only regular, normalized JSON data can make it into the system.

The rest of the chapter has us implement a couple practical programs, such as:

fetching JSON-formatted COVID-19 statistics from Japanese government sites, and displaying the results
parsing JSON on the command-line to get runtime program arguments (and displaying a nice message when the runtime JSON parsing fails)

Some thoughts

It took a long time for me to publish this post — I started implementing the code in this chapter on October 28th, and finished the exercises over the Christmas holiday break.

In between when I started and now, the Japansese COVID statistics website https://data.corona.go.jp has gone down. When the COVID pandemic was happening, it seemed to be the most important thing in the world, so much that a university textbook used a goverment stats website as a permanent reference. ~~Now, the data has been moved to the website of the Japanese Health Ministry, and is only available as CSV.~~

~~As a result of this, subsequent chapters (SQL integration, etc.) will need some tweaking, as they all rely on the JSON data source being up. In either case, stay tuned for more in the coming year.~~

2024-01-02 Update

Luckily, the authors of the book have preserved a copy of the reference data in a github repo with example code.

Finally, thank you for reading this blog. May you have a Happy New Year 2024!

Python defaultdict as sparse array: use with caution

Sun, 10 Dec 2023 00:00:00 +0100

TLDR: When using defaultdicts to represent sparse data with large cardinalities, remember that reading values with [] is side-effecting and increases the size of the dictionary.

Use .get() for reads, and reserve [] for insertions, increments, and updates.

Python’s collections package has some nice convenient datastructures that you might be familiar with, such as namedtuple or deque.

There is one very interesting, but flawed datastructure in there, namely the defaultdict. The defaultdict lets you, the programmer, define a ‘default factory’ lambda, which will be called to create (and persist) a default value in the dictionary, when __getitem__ (i.e. []-access) is called.

Here’s an example:

>>> from collections import defaultdict
>>> d = defaultdict(lambda: list())
>>> d["animals"]
[]
>>> d["animals"].append("dog")
>>> d
defaultdict(<function <lambda> at 0x7f53579c8860>, {'animals': ['dog']})

`defaultdict` and sparse arrays

Syntactically, this is very nice when you’re creating or updating sparse arrays, because you can just “fire and forget”, setting and incrementing (or decrementing) values at any index you can think of:

>>> d = defaultdict(lambda: 0)
>>> d[0] = 1
>>> d[100] += 5
>>> d[100] += 5
>>> d[100]
10
>>> d[99]
0

However, this type of syntactic convenience leads to complacency. __getitem__ access on a defaultdict is not the equivalent of .get() with a nice default. It is, unfortunately a side-effecting operation, as the value produced by the default factory is stored in the dict before being returned.

Now, it does say so right in the documentation:

If default_factory is not None, it is called without arguments to provide a default value for the given key, this value is inserted in the dictionary for the key, and returned.

The large-scale implications of this behavior are often not evident until it turns out your code is somehow performing very slowly and consuming lots of memory.

`defaultdict` effectively un-sparsifies your sparse array if you’re querying for every value

Here is a demonstration of the side-effecting behavior in action.

# file: sparse.py
import sys
from collections import defaultdict

def test(use_get):
    n = 1000*1000*10
    sparse = defaultdict(lambda: 0)
    sparse[int(n/3)] = 1
    sparse[int(n/3)*2] = 2
    sparse[int(n/3)*3] = 3
    print("SIZE BEFORE:", sys.getsizeof(sparse))
    sum = 0
    if use_get:
        # We use .get() so as not to trigger the default_factory
        for idx in range(n):
            sum += sparse.get(idx, 0)
    else:
        # We use []-access, bloating the dict as a side-effect
        for idx in range(n):
            sum += sparse[idx]
    print("SIZE AFTER:", sys.getsizeof(sparse))
    print(f"RESULT (using get: {use_get})", sum)

if sys.argv[1] == "True":
    test(True)
else:
    test(False)

In the code above, we first create a dictionary representing a sparse vector with 10-million entries, the default value being 0. Then, we set up some actual values roughly at every 1/3rd of the vector length. Our goal is to sum up all the values in this sparse array, going through every index.

If use_get is set to True, we use the .get() method on our defaultdict. If use_get is set to False, we use []-access.

We measure the size of the dict before and after the summation. Here are the results and the runtimes:

# Using .get() method:
/bin/time python3 ./sparse.py True
SIZE BEFORE: 232
SIZE AFTER: 232
RESULT (using get: True) 6
1.20user 0.00system 0:01.21elapsed 99%CPU (0avgtext+0avgdata 9984maxresident)k

# Using []-syntax:
/bin/time python3 ./sparse.py False
SIZE BEFORE: 232
SIZE AFTER: 335544408
RESULT (using get: False) 6
3.33user 0.82system 0:04.17elapsed 99%CPU (0avgtext+0avgdata 677260maxresident)k

Much worse! The dict ballooned from 232 bytes to over 300 megabytes. Additionally, it took the python runtime almost three times longer to process all 20 million keys in our sparse vector.

When using []-notation with a defaultdict, one must always remember to only use it for side-effecting operations, where updating the dictionary is in fact called for. If the goal is just to read a value, .get() with a sensible default is the way to go.

NuScratch: Getting around the 'Unknown class: SmallFloat64' error

Sat, 02 Dec 2023 00:00:00 +0100

Polish schoolchildren learn the basics of coding in the MIT Scratch creative programming environment. I was very happy when my oldest daughter really got into programming and asked me if we can “have Scratch at home”.

But we do have Scratch at home, I told her.

Scratch at home (when trying to save a project):

Jokes aside, I was quite frustrated because NuScratch, the continuation of the original Scratch, simply failed to save any coding projects created within it.

I kept getting the same error on all three Linux machines at home, pretty much regardless of which combination of (Squeak image × NuScratch version) I tried. The same error kept popping up on all our household machines: a void linux box, a devuan box, and on the raspberry pi os. Even though NuScratch was installed by default on the Raspberry Pi as part of the “recommended software” bundle!

It took me a week of debug attempts, trying to figure out how NuScratch operates and how the GUI is put together in Squeak Smalltalk. I had a lot of fun and learned quite a bit about the Smalltalk ecosystem. When I finally got to the failing code/tests in question (the ObjStream class cannot write files), I realized that the problem lies in deep 32-bit assumptions made in the original Scratch serialization protocol.

So, since googling for the solution failed to surface any hints, here is the solution.

Q: My Scratch or NuScratch system errors out and displays the following pop-up when I try to save my project:

Save failed:
Error: Unknown class SmallFloat64.

A: Make sure you are running a 32-bit Squeak VM on a 32-bit operating system. On Raspberry Pi, this will be something like 2023-10-10-raspios-bookworm-armhf-full.

Working Review of "Practical ML Programming with SML#" (Ohori, Ueno), CHAPTER 7

Sun, 22 Oct 2023 00:00:00 +0200

Interoperability with the C language

I was quite ambivalent going into the chapter on C interoperability. For one, I’ve never done any serious programming in C, and two, I know that C FFIs are notorious for being difficult to operate. Having done the exercises, I can say that SML# does live up to its claims of easy C interoperability, but there are some papercuts that the C-naïve programmer (such as yours truly) will have to sustain to make it all come together.

First, let’s quickly summarize the contents of this chapter. Then, below, I’ll list out my various gripes and difficulties, including my methods of overcoming them.

Chapter summary

7.1 ML and the role of C interoperability

A quick motivating summary of why C interop is important. The authors make some very good points about hardware-specific libraries becoming first available as C libraries. This intro serves to whet your appetite for the subsequent tech demos.

7.2 Datatypes suitable for direct pass-through

Starting out with the easy things first – in this case, a list of SML# datatypes that can be directly passed to C functions as arguments and received as return values. Unsurprisingly, we’re limited to chars and the numeric types, excluding IntInfs (a.k.a BigInts).

7.3 C import expressions

A short and sweet section that has us define a little C library, and then import it directly into the interactive smlsharp interpreter, utilizing the numeric types described in the previous section. Overall, the didactic value would be excellent, if not for the fact that the example won’t work without explicitly specifying LD_LIBRARY_PATH on the command-line. This is a pattern that will repeat itself, and on which I’ll have more to say, below.

7.4 Separate compilation and linking

Kicking it up a notch, we’re now going to develop an SML# module that encapsulates a C library. In this case, it’s the Mersenne Twister random number generator, another Japanese contribution to software engineering. Excellent.

This chapter feels like a “production-grade” restatement of 7.3. It exposes the entire surface area of the Mersenne Twister C library, and demonstrates how to modify the autogenerated Makefile to ensure that the .so file is linked to the output binary.

I did have several difficulties completing this one, ranging from the innocent (the URL for the MT source files has changed at Hiroshima University), to the annoying (had to modify the autogenerated Makefile by hand to include the .so file), and the already known (need to set up LD_LIBRARY_PATH in order to run the resulting binary).

Overall, the difficulties are not insurmountable, and once the kinks are ironed out, it’s quite a fun experience to build a Mersenne Twister facade structure in SML#. This chapter does get a bit of treatment below.

7.5 Exporting data with a pointer-based runtime representation

The datatypes in question are: string, τ ref, τ array, tuples, and records. As long as the τ is one of the types from section 7.2, we can freely pass these in to C functions as-is. The caveat here is that strings are interpreted as const char *, as they are immutable in SML#.

The example used to illustrate this is modf from the C standard library, which overwrites its second argument (double *iptr in C, real ref in SML#). While its easy to work with simple types, I think marshaling SML# records into C-land, with their fields being order-dependent, would be a more involved topic.

7.6 Importing data with a pointer-based runtime representation

While translating pointer-based SML# data into C is automatically done by the SML# runtime, we can’t create such data in C-land and then use it on the SML# side. This is because pointers created in C lack the necessary GC metadata for the SML# runtime to be able to handle their lifecycle correctly.

Hence, it’s necessary to export the data from C as pointers, and subsequently read the pointed-to data from SML#, using the Pointer structure.

Apart from listing the interface of structure Pointer, this section shows us how to read strings into SML# (via getenv), and also how to use FILE pointers to do byte-based disk I/O. Fun stuff, no snags.

7.7 An example of integrating a polymorphic C library

The famous quicksort is up next, but we’re not going to be rehashing the old “quicksort in three lines” trope of functional programming. We’re actually going to implement a polymorphic (and in-place mutating) quicksort in SML#, which will use, under the hood, the stdlib qsort:

void qsort(void *base, size_t nmemb, size_t size, int (*compar)(const void *, const void *));

This is a much more advanced example, which features a polymorphic subject pointer (base), the need to know the size of an array element in C (size_t size), and requires us to pass in a pointer to a comparison function.

This section is probably the most satisfying of all. An elegant implementation of a complex FFI interaction is much more impressive than the examples involving marshaling numbers across the C boundary.

In the end, we have a typesafe qsort function which mutates arrays in-place, an which we can use to sort the numbers conveniently generated with the Mersenne Twister from the beginning of the chapter.

7.8 Exercises

After having done the exercises, we’ll have a typesafe quicksort which accepts the ‘standard’ SML comparison functions, as defined in the Basis library.

qsort inputArray Int64.compare

Now, the gripes

1. `LD_LIBRARY_PATH`

Maybe I’m missing something here, and it’s my lack of C experience speaking, but the SML# tooling simply doesn’t work with directory-local .so files. Despite the examples all using -L. -lmy to load libmy.so, this simply doesn’t work on my machine. It’s not a big deal for me and doesn’t detract from the experience, buy I can’t help but feel that there’s something missing here: either in my knowledge of C-based workflows, or in the difference between my machine and the SML# creators’ machines.

My solution is as follows (using libsqr.so as an example)

% cat libsqr.c
short sqrShort (short n) { return (n * n); }

% gcc -shared -fPIC libsqr.so libsqr.c

% smlsharp -L. -lsqr
SML#  for x86_64-unknown-linux-gnu with LLVM 12.0.1
# val sqr = _import "sqrShort" : int16 -> int16;
dynamic link failed: libsqr.so: cannot open shared object file: No such file or directory

% LD_LIBRARY_PATH=$(pwd) smlsharp -L. -lsqr
SML#  for x86_64-unknown-linux-gnu with LLVM 12.0.1
# val sqr = _import "sqrShort" : int16 -> int16;
val sqr = fn : int16 -> int16
# sqr 16;
val it = 256 : int16

2. Autogenerated Makefiles and C library compilation

I love the fact that SML# leans on make to achieve its separate compilation capabilities. But I’ve found I disagree with the designers on the role of the autogenerated Makefiles in the general programming flow.

In my personal workflows (including this blog), I like to use Makefiles as a top-level driver for development. So, for example, to build and publish the Jekyll output, I’ll run make publish, and that’s that.

When developing a multi-language project, it makes sense to tie together the various build tools offered by the languages (npm, mix, cargo) at a high level, so that running make build builds all the subcomponents of a program, farming out the language-specific details to, say mix compile, etc.

So what happens when we’re developing a C-based library inside of our SML# project, and we want to use a top-level Makefile to run it all?

So far, I’ve used this pattern:

all0: Makefile.smlsharp all

Makefile.smlsharp: $(shell find . | grep '.smi$$')
	smlsharp -MMm main.smi > $@

include Makefile.smlsharp

This lets me have one standard Makefile on the top-level of my project, where I could define non-SML# targets and tasks, and also have smlsharp regenerate Makefile.smlsharp whenever there are changes in the inter-module dependency structures.

In short, Makefile.smlsharp is fully defined by the existing .smi files in the project, hence I think of it as non-essential, ephemeral data that can go into .gitignore. I don’t ever have to look at these files, I just rely on the fact that they can build main along with all its dependencies.

Now, if we add a C-based shared library into our development mix, we have something like the following:

all0: mt19937-64.so Makefile.smlsharp all

mt19937-64.so: mt19937-64.c
	gcc -shared -fPIC $< -o $@

mt19937-64.c:
	curl -s -O http://www.math.sci.hiroshima-u.ac.jp/m-mat/MT/VERSIONS/C-LANG/mt19937-64.c

Makefile.smlsharp: $(shell find . | grep '.smi$$')
	smlsharp -MMm main.smi > $@

include Makefile.smlsharp

But now, our main file will fail to link to mt19936-64.so. Why? Because we need to hand-edit the autogenerated Makefile and change the line

LIBS =

LIBS = mt19937-64.so

This requirement effectively changes the status of the autogenerated Makefile from ephemeral and easily-recreated to something non-ephemeral, requiring frequent manual modification.

Since I honestly believe we should be able to treat the autogenerated Makefiles as throwaway artifacts, I’ve made a pull request to SML# that, if accepted, will allow us to define LIBS in the environment, and have the autogenerated Makefile pick it up.

This means that we can put the line

LIBS=mt19937-64.so

in our top-level Makefile and never have to hand-edit the output of smlsharp -Mmm.

3. The Pointer structure

All of the examples involving marshaling data into SML# involve the interactive interpreter, and not standalone compilation. So when the time comes to compile your binary with quicksort, you run into this error message:

  (name evaluation "190") unbound variable: Pointer.load

Using the very same files in an interactive session works, but compilation fails. Poring over the example code in the chapter gives no clue.

It turns out that you need to _require "ffi.smi" to get access to the pointer structure. It helps to have the SML# source checked out locally, as the example code is very helpful in figuring out small things like this.

I feel that the lack of _require "ffi.smi" in the chapter text is an omission.

4. The language of import declarations

This gripe is perhaps very minor, but it’s details like this that make some languages much harder to learn than others. In Standard ML—and by extension in SML#–there are two ‘languages’ to master. One is the type-level language, the other is the value-level language. They are similar but not the same.

For example, the type-level definition of a function that takes a 2-tuple of two types and returns nothing is:

(* Type language *)
val someFun : 'a * 'b ref -> unit

Now, the value-level implementation of this has a slightly different syntax:

(* Value language *)
fun someFun (a,b) = ()

Note the differences:

1) The type-level tuple constructor is _ * _. The value-level tuple constructor is ( _ , _ ).

2) The type-level syntax for a type variable is 'a. The value-level variable is a.

3) The name of the unit type is unit. The value for the unit type is ().

On top of this confusion, SML# throws in another language: the language describing the types of imported FFI functions. For the function above, it would be something like:

val c_someFun = _import "someFun" : ('a, 'b ptr) -> ()

This language seems to combine both the value-level and type-level languages. I know there must have been a good reason for introducing it, but I feel having fresh learners learn yet another subtle “DSL” is an educational barrier.

For what it’s worth, Haskell does a better job of having the type-level and value-level languages resemble each other to a greater degree:

-- type language
someFun :: (a, b) -> ()

-- value language
someFun (a,b) = ()

Summing up

This chapter was very interesting to read and implement. For someone who is not used to working with C, there are some gotchas that aren’t clearly marked in the book, and I had to figure things out myself.

As with all things related to binary interoperability, it feels like a lot of the designs were informed by real-world constraints and tastes of the authors.

All-in-all, it feels like developing an SML# wrapper around C libraries should be quite straightforward. This bodes well for future chapters, which involve graphics programming with Cairo and the like. I’m looking forward to what comes next.