Moved over first rbtree test from koka_bench and integrated Kraken via compiler wrapper script that calls the partial_evaluator / compiler and then emits a wrapper script that runs the resulting wasm via wasmtime.
This commit is contained in:
Nathan Braswell
131
koka_bench/README.md
Normal file
131
koka_bench/README.md
Normal file
@@ -0,0 +1,131 @@
|
||||
# NOTE
|
||||
|
||||
These are the benchmarks extracted from the Koka repository https://github.com/koka-lang/koka
|
||||
at b1670308f88dd1fc6c22cad28385fcb185d5b27d from the test/bench directory
|
||||
and modified by me to run inside a reproducable environment defined by a Nix Flake,
|
||||
and to use Bash to coordinate and Hyperfine to time instead of the Koka based runner.
|
||||
|
||||
./test.sh will build and run the benchmarks.
|
||||
|
||||
I'm slowly porting them over to the Kraken project, and adding Kraken versions of each one.
|
||||
|
||||
|
||||
## Some explanation from a PLDI Readme originally included:
|
||||
|
||||
Available languages are:
|
||||
|
||||
- `kk` : Koka v2.1.1 compiling using gcc 9.3.0.
|
||||
- `kkx` : Koka v2.1.1 compiling using gcc 9.3.0 but without reuse optimization (Section 2.4).
|
||||
- `ml` : OCaml v4.08.1 using the optimizing compiler (`ocamlopt`)
|
||||
- `hs` : Haskell GHC 8.6.5.
|
||||
- `sw` : Swift 5.3.3.
|
||||
- `jv` : Java 15.0.2, Java(TM) SE Runtime Environment (build 15.0.2+7-27),
|
||||
Java HotSpot(TM) 64-Bit Server VM (build 15.0.2+7-27, mixed mode, sharing).
|
||||
- `cpp` : GCC 9.3.0,
|
||||
|
||||
Available tests are described in detail in Section 4 and are:
|
||||
|
||||
- `rbtree` : inserts 42 million items into a red-black tree.
|
||||
- `rbtree-ck` : a variant of rbtree that keeps a list of every 5th subtree and thus shares many subtrees.
|
||||
- `deriv` : the symbolic derivative of a large expression.
|
||||
- `nqueens` : calculates all solutions for the n-queens problem of size 13 into a list, and returns the length of that list.
|
||||
- `cfold` : constant-folding over a large symbolic expression.
|
||||
|
||||
|
||||
# Original README follows:
|
||||
|
||||
|
||||
# Build and run benchmarks
|
||||
|
||||
This contains the standard benchmark suite (discussed in detail in [Perceus] paper).
|
||||
It is still basic but more benchmarks
|
||||
with effect handlers are coming. The suite can run on (Ubuntu Linux), WSL2, and macOS,
|
||||
and the benchmarks need:
|
||||
|
||||
- `gcc`. Should be there, otherwise use `sudo apt install gcc`,
|
||||
- `ghc`. Use `sudo apt install ghc`,
|
||||
- `ocamlopt`. Use `sudo apt install ocaml`.
|
||||
We used the new multi-core OCaml, see <https://github.com/ocaml-multicore/multicore-opam>
|
||||
for installation instructions (including `domainslib` for the binarytrees benchmark)
|
||||
```
|
||||
> opam update
|
||||
> opam switch create 4.12.0+domains+effects --repositories=multicore=git+https://github.com/ocaml-multicore/multicore-opam.git,default
|
||||
> opam install dune domainslib
|
||||
```
|
||||
|
||||
- `swiftc`. The Swift compiler can be downloaded [here](https://swift.org/download/).
|
||||
The benchmarks expect `swiftc` to be installed at `/opt/swift/bin`,
|
||||
so unpack and copy everything under `swift-.../usr` to `/opt/swift/bin`:
|
||||
```
|
||||
> tar -xzf swift-5.5-RELEASE-ubuntu20.04.tar.gz
|
||||
> cd swift-5.5-RELEASE-ubuntu20.04/usr
|
||||
> sudo mkdir /opt/swift
|
||||
> sudo cp -r * /opt/swift
|
||||
```
|
||||
|
||||
- `javac`/`java`. We used these [instructions](https://www.linuxcapable.com/how-to-install-java-17-lts-jdk-17-on-ubuntu-20-04/)
|
||||
to install the Java SE 17 Hotspot compiler:
|
||||
```
|
||||
> sudo apt update
|
||||
> sudo add-apt-repository ppa:linuxuprising/java
|
||||
> sudo apt-get -y install oracle-java17-installer oracle-java17-set-default
|
||||
> java --version
|
||||
java 17 2021-09-14 LTS
|
||||
Java(TM) SE Runtime Environment (build 17+35-LTS-2724)
|
||||
Java HotSpot(TM) 64-Bit Server VM (build 17+35-LTS-2724, mixed mode, sharing)
|
||||
```
|
||||
|
||||
The benchmarks can now be build using:
|
||||
|
||||
```
|
||||
> cd test/bench
|
||||
> mkdir build
|
||||
> cd build
|
||||
> cmake .. -DCMAKE_BUILD_TYPE=Release
|
||||
> cmake --build .
|
||||
```
|
||||
|
||||
For some benchmarks, like `cfold`, we may need a large stack, so it may be good to raise the limit:
|
||||
```
|
||||
> ulimit -s unlimited
|
||||
```
|
||||
|
||||
We can then run all benchmarks as:
|
||||
```
|
||||
> ctest .
|
||||
```
|
||||
Or only run benchmarks for one language with `-L <lang>`:
|
||||
```
|
||||
> ctest -L koka
|
||||
```
|
||||
Or run specific benchmarks using `-R <regex>`,
|
||||
like the symbolic derivative benchmark:
|
||||
```
|
||||
> ctest -R deriv
|
||||
Test project /home/daan/dev/koka/test/bench/build
|
||||
Start 4: hs-deriv
|
||||
1/4 Test #4: hs-deriv ......................... Passed 2.29 sec
|
||||
Start 10: kk-deriv
|
||||
2/4 Test #10: kk-deriv ......................... Passed 1.25 sec
|
||||
Start 19: ml-deriv
|
||||
3/4 Test #19: ml-deriv ......................... Passed 1.73 sec
|
||||
Start 25: sw-deriv
|
||||
4/4 Test #25: sw-deriv ......................... Passed 2.88 sec
|
||||
|
||||
100% tests passed, 0 tests failed out of 4
|
||||
...
|
||||
```
|
||||
|
||||
We can also run the tests using the `test/bench/bench.kk` script instead of
|
||||
using `ctest` which also measures peak working set and calculates
|
||||
normalized scores. For example, from the `build` directory, we can run all benchmarks as:
|
||||
```
|
||||
> koka -e ../bench
|
||||
```
|
||||
Use the `--lang` or `--test` options to specify a comma separated list of
|
||||
languages or benchmarks:
|
||||
```
|
||||
> koka -e ../bench -- --lang=koka,ocaml --test=rbtree,rbtree-ck
|
||||
```
|
||||
The `-i<N>` switch runs `N` iterations on each benchmark and calculates
|
||||
the average and the error interval.
|
||||
Reference in New Issue
Block a user