# NOTE

These are the benchmarks extracted from the Koka repository https://github.com/koka-lang/koka
at b1670308f88dd1fc6c22cad28385fcb185d5b27d from the test/bench directory
and modified by me to run inside a reproducable environment defined by a Nix Flake,
and to use Bash to coordinate and Hyperfine to time instead of the Koka based runner.

**It is under the Apache 2 License to match the source Koka, not BSD-2-Clause-Patent of the rest of Kraken**

./test.sh will build and run the benchmarks.

I'm slowly porting them over to the Kraken project, and adding Kraken versions of each one.


## Some explanation from a PLDI Readme originally included:

Available languages are:

- `kk`  : Koka v2.1.1 compiling using gcc 9.3.0.
- `kkx` : Koka v2.1.1 compiling using gcc 9.3.0 but without reuse optimization (Section 2.4).
- `ml`  : OCaml v4.08.1 using the optimizing compiler (`ocamlopt`)
- `hs`  : Haskell GHC 8.6.5.
- `sw`  : Swift 5.3.3.
- `jv`  : Java 15.0.2, Java(TM) SE Runtime Environment (build 15.0.2+7-27), 
          Java HotSpot(TM) 64-Bit Server VM (build 15.0.2+7-27, mixed mode, sharing).
- `cpp` : GCC 9.3.0, 

Available tests are described in detail in Section 4 and are:

- `rbtree`    : inserts 42 million items into a red-black tree.
- `rbtree-ck` : a variant of rbtree that keeps a list of every 5th subtree and thus shares many subtrees.
- `deriv`     : the symbolic derivative of a large expression.
- `nqueens`   : calculates all solutions for the n-queens problem of size 13 into a list, and returns the length of that list.  
- `cfold`     : constant-folding over a large symbolic expression.


# Original README follows:


# Build and run benchmarks

This contains the standard benchmark suite (discussed in detail in [Perceus] paper).
It is still basic but more benchmarks
with effect handlers are coming. The suite can run on (Ubuntu Linux), WSL2, and macOS,
and the benchmarks need:

- `gcc`. Should be there, otherwise use `sudo apt install gcc`,
- `ghc`. Use `sudo apt install ghc`,
- `ocamlopt`. Use `sudo apt install ocaml`.
  We used the new multi-core OCaml, see <https://github.com/ocaml-multicore/multicore-opam> 
  for installation instructions (including `domainslib` for the binarytrees benchmark)
  ```
  > opam update
  > opam switch create 4.12.0+domains+effects --repositories=multicore=git+https://github.com/ocaml-multicore/multicore-opam.git,default
  > opam install dune domainslib
  ```

- `swiftc`. The Swift compiler can be downloaded [here](https://swift.org/download/).
   The benchmarks expect `swiftc` to be installed at `/opt/swift/bin`,
   so unpack and copy everything under `swift-.../usr` to `/opt/swift/bin`:
   ```
   > tar -xzf swift-5.5-RELEASE-ubuntu20.04.tar.gz
   > cd swift-5.5-RELEASE-ubuntu20.04/usr
   > sudo mkdir /opt/swift
   > sudo cp -r * /opt/swift
   ```

- `javac`/`java`. We used these [instructions](https://www.linuxcapable.com/how-to-install-java-17-lts-jdk-17-on-ubuntu-20-04/)
   to install the Java SE 17 Hotspot compiler:
   ```
   > sudo apt update
   > sudo add-apt-repository ppa:linuxuprising/java
   > sudo apt-get -y install oracle-java17-installer oracle-java17-set-default
   > java --version
   java 17 2021-09-14 LTS
   Java(TM) SE Runtime Environment (build 17+35-LTS-2724)
   Java HotSpot(TM) 64-Bit Server VM (build 17+35-LTS-2724, mixed mode, sharing)
   ```

The benchmarks can now be build using:

```
> cd test/bench
> mkdir build
> cd build
> cmake .. -DCMAKE_BUILD_TYPE=Release
> cmake --build .
```

For some benchmarks, like `cfold`, we may need a large stack, so it may be good to raise the limit:
```
> ulimit -s unlimited
```

We can then run all benchmarks as:
```
> ctest .
```
Or only run benchmarks for one language with `-L <lang>`:
```
> ctest -L koka
```
Or run specific benchmarks using `-R <regex>`,
like the symbolic derivative benchmark:
```
> ctest -R deriv      
Test project /home/daan/dev/koka/test/bench/build
    Start  4: hs-deriv
1/4 Test  #4: hs-deriv .........................   Passed    2.29 sec
    Start 10: kk-deriv
2/4 Test #10: kk-deriv .........................   Passed    1.25 sec
    Start 19: ml-deriv
3/4 Test #19: ml-deriv .........................   Passed    1.73 sec
    Start 25: sw-deriv
4/4 Test #25: sw-deriv .........................   Passed    2.88 sec

100% tests passed, 0 tests failed out of 4
...
```

We can also run the tests using the `test/bench/bench.kk` script instead of
using `ctest` which also measures peak working set and calculates
normalized scores. For example, from the `build` directory, we can run all benchmarks as:
```
> koka -e ../bench
```
Use the `--lang` or `--test` options to specify a comma separated list of
languages or benchmarks:
```
> koka -e ../bench -- --lang=koka,ocaml  --test=rbtree,rbtree-ck
```
The `-i<N>` switch runs `N` iterations on each benchmark and calculates
the average and the error interval.