Hacking on iota

Table of Contents

↑ Back to the index

1. Toolchain

Any GHC from 9.4 onwards. On Arch: 

sudo pacman -S ghc cabal-install

Or with ghcup, which is ghcup-hs-bin on the AUR: 

ghcup install ghc recommended
ghcup install cabal latest

Then: 

cabal build
cabal test
cabal repl

The Makefile at the top wraps the same commands: make build, make test, make docs for the HTML documentation and make clean. make on its own does the first three. CABALFLAGS is passed on to cabal, as in make build CABALFLAGS=–ghc-options=-Werror. make clean leaves the checked-in HTML alone.

hie.yaml is checked in, so haskell-language-server finds the project on its own.

Building the HTML documentation also needs Emacs. See Documentation.

2. Where things go

Change Where
lexical syntax src/Iota/Reader.hs and Test.Reader
a new core form special in src/Iota/Expand.hs and specialForms in Expand/Common.hs
a new derived form rewrite in src/Iota/Expand/Derived.hs, in the style of chapter 7.3
libraries, import, cond-expand src/Iota/Expand/Library.hs
a procedure that needs Haskell the matching module under src/Iota/Primitives/
a procedure that calls back into Scheme a PrimitiveK in Iota.Primitives.Control, or write it in Scheme
a procedure expressible in Scheme lib/iota/stdlib.iota, which is embedded at compile time, so rebuild
exporting it from a standard library the Appendix A lists in src/Iota/Library.hs
printing src/Iota/Printer.hs and Test.Printer
numerics, complex numbers src/Iota/Number.hs and Test.Number
syntax-rules src/Iota/SyntaxRules.hs and Test.Macro
evaluation and continuations src/Iota/Eval.hs

When in doubt, write it in Scheme. A primitive that calls a Scheme procedure has to be a PrimitiveK and must not throw for Scheme level errors, which means using guardIO, and it is easy to get that wrong in a way the tests will not catch. Anything written in Scheme gets continuations and dynamic-wind right without trying.

A name only becomes visible to programs once it appears in an export list in Iota.Library. If a listed name is not defined, startup fails immediately rather than at first use.

3. Tests

cabal test runs one tasty tree:

  1. Unit tests for the pure parts, meaning the reader, the printer and the numbers. Unit tests for syntax-rules in Test.Macro, driven with a hand-built syntactic environment. QuickCheck properties in Test.Property for read and write round trips and number formatting.
  2. Evaluator cases in test/Test/Eval.hs and Scheme.hs. Each is an expression and its expected write output, run in a fresh interpreter.
  3. File and process tests in Test.Ports. File ports, libraries loaded from disk, include, error locations, exit.
  4. Golden tests. Every test/scheme/NAME.iota is run and its output, meaning everything it displayed followed by the written value of its last form, is compared against test/golden/NAME.golden. This is the test and test/ans layout from adamwespiser/scheme. Output is captured by pushing a string port onto the current-output-port parameter, see Iota.Interpreter.captureOutput.

To add a golden test: 

echo '(+ 1 2)' > test/scheme/my-case.iota
cabal run iota -- run test/scheme/my-case.iota   # check it by eye
cabal test --test-options='--accept'             # writes the .golden

The accept flag rewrites every golden file from the current output. Read the diff before committing.

4. Conventions

  • -Wall stays clean. Build with -Werror before sending a change.
  • Language extensions go in the cabal file, not in pragmas.
  • A helper lives in the module that owns the type, not where it was first needed.
  • Comments say why, not what. If the code needs a comment to say what it does, the code is the problem.
  • The report wins. When in doubt, read the relevant section of R7RS and do what it says. A deliberate deviation goes on the list at the end of docs/r7rs-coverage.org, with the reason.

5. Sending a change

Open a pull request against main. Keep it to one thing. A fix comes with a test that fails without it. If the change is user visible, update docs/iota.org or docs/r7rs-coverage.org in the same change. Nothing is merged with a failing test suite.

Bug reports are best with the smallest program that shows the problem, the output you expected, and the output of iota –version.

6. Documentation

The pages under docs/ are Org files, exported to HTML next to their source by Emacs in batch mode. Emacs is the only tool this needs on top of the toolchain above. On Arch: 

sudo pacman -S emacs

Then, from the top of the repository: 

make docs            # export every page whose .org file changed
make -C docs ids     # give new headings a CUSTOM_ID, see below

and open docs/index.html in a browser. A change to setup.org or export.el rebuilds every page. EMACS=/path/to/emacs picks another Emacs than the one on the PATH.

GitHub Pages serves docs/ as it is, so the HTML is checked in. Rebuild it and commit it in the same change as the .org file it comes from. The HTML in the repository was exported with Emacs 31.1 and the Org 9.8.7 that comes with it. Another Org version may write slightly different HTML, and then every page shows up as changed.

docs/export.el is what the Makefile loads into Emacs. It makes the export reproducible: an unchanged page gives an unchanged file.

Every heading has a CUSTOM_ID property, which becomes its id in the HTML and the target of the ¶ link that shows when you hover over it. The id stays put when the heading is renamed, so links to a section keep working. make -C docs ids gives each new heading one, made from its text. Until then the export derives the same id on the fly, but it follows the heading text.

docs/setup.org holds what every page shares: the export options, the stylesheet, and the Prism scripts that highlight source blocks in the browser. Pages pull it in with #+SETUPFILE: setup.org. A new page needs that line, a link back to index.org, an entry in the list in index.org, and a line under extra-doc-files in iota.cabal.

The Prism files come from cdnjs with integrity hashes. A source block in a new language needs its Prism component added to setup.org, and bumping a version means new hashes: 

curl -s URL | openssl dgst -sha384 -binary | openssl base64 -A

7. Debugging

A few things that have saved me time:

  • The repl's :ast command shows what the reader made of an expression. Handy when a macro seems to see the wrong shape.
  • cabal repl, then import Iota, then evalText "(+ 1 2)" gives you the interpreter from ghci. newInterpreter and runText do the same with a persistent environment.
  • A test in Test.Eval is one line. Write the failing case there first and work backwards from it.
  • Test.Macro drives syntax-rules with a hand built environment, which is the quickest way to poke at a hygiene problem.

8. Checking a build by hand

Before a release, or after anything that touches the build, run the same steps on the GHC versions listed under tested-with in iota.cabal: 

cabal build all --enable-tests --ghc-options=-Werror
cabal test
cabal haddock lib:iota
for f in examples/*.iota; do cabal run -v0 iota -- "$f"; done

ghcup makes switching GHC versions painless.

9. Cutting a release

  1. The checks above, on every GHC in tested-with.
  2. cabal haddock, and read it once for anything that looks wrong.
  3. cabal check and cabal sdist. The tarball has to build on its own.
  4. Make sure docs/r7rs-coverage.org still matches what the code does, and that make -C docs html still exports every page.
  5. Bump the version in iota.cabal, following the PVP.
  6. Tag it, vX.Y, and push the tag.