miloignis/kraken
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
c3b2a852b76ec4dd3b65dce91a6281d31f82b9b0
kraken/fib_tests.sh

136 lines
5.4 KiB
Bash
Raw Normal View History

Add fib_let test and script to run all of them. Indeed, the version with the let is worse when interpreted and the same/better when partially-evaled and compiled
2022-03-20 16:01:38 -04:00
#!/usr/bin/env bash
Fixed a terrible bug where turns out I used the same name for a block and a parameter in the comparison meta-function - they share the same namespace in the wasm DSL, so when I used it like a parameter in a loop it resolved to the number of scopes between the statement and the block'th parameter which had the same type and the calculation worked fine, but it overwrote the parameter I thought wasn't being used and called a function later with. Also, that seemed like that last of the really bad leaks & corruption, so re-enabling the freelist and am able to run some less-trivial benchmarks, namely (fib 30)! The compiled version is between 4.7x and 65x slower than Python, depending on if you're using wasmer, wasmtime, wasm3. Would like to try WAVM as well. A solid place to start, I think, and hopefully we'll be faster than Python once I implement a variety of dont-be-dumb optimizations (real-er malloc, static calls of statically known functions, etc)
2022-03-29 23:49:51 -04:00
NUMBER=30
Round allocated blocks up to the nearest 8 words, and split blocks that are >= 2x needed number of words. Now only allocates 1 wasm page for both compiled and interpreted versions at fib 30, a 269-538x improvement!
2022-03-30 21:27:01 -04:00
#NUMBER=25
Add fib_let test and script to run all of them. Indeed, the version with the let is worse when interpreted and the same/better when partially-evaled and compiled
2022-03-20 16:01:38 -04:00
Use hyperfine to benchmark, add builtin_fib as a comparison for how fast we could try to be
2022-04-13 00:25:53 -04:00
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib.kp
mv csc_out.wasm fib_compiled.wasm
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib_let.kp
mv csc_out.wasm fib_compiled_let.wasm
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib.kp no_compile
mv csc_out.wasm fib_interpreted.wasm
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib_let.kp no_compile
mv csc_out.wasm fib_interpreted_let.wasm
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib2.kp
mv csc_out.wasm fib_compiled_manual.wasm
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm builtin_fib.kp
mv csc_out.wasm builtin_fib.wasm
pushd rust_fib
cargo build --target=wasm32-wasi
cargo build --release --target=wasm32-wasi
cargo build
cargo build --release
popd
Add inlining of add and subtract, and now might be beating Python, though not by a statistically significant amount with the number of tests. Fib is still allocating 4 million times or so, which is weird, since +&- should have been the last calls to do so. Time to track that down
2022-04-20 23:47:36 -04:00
#pushd rust_let
#cargo build --target=wasm32-wasi
#cargo build --release --target=wasm32-wasi
#cargo build
#cargo build --release
#popd
#
#pushd clojure_fib
#lein uberjar
#popd
#
#pushd clojure_hi
#lein uberjar
#popd
Use hyperfine to benchmark, add builtin_fib as a comparison for how fast we could try to be
2022-04-13 00:25:53 -04:00
#clang-11 fib.c -o c_fib
#clang-11 fib_let.c -o c_fib_let
Add inlining of add and subtract, and now might be beating Python, though not by a statistically significant amount with the number of tests. Fib is still allocating 4 million times or so, which is weird, since +&- should have been the last calls to do so. Time to track that down
2022-04-20 23:47:36 -04:00
hyperfine --warmup 2 --export-markdown table.md \
Use hyperfine to benchmark, add builtin_fib as a comparison for how fast we could try to be
2022-04-13 00:25:53 -04:00
'echo '$NUMBER' | wasmtime ./fib_compiled.wasm' 'echo '$NUMBER' | wasmtime ./fib_compiled_let.wasm' \
Fix regression - was using the wrong error when blocking recursion and treating as real error instead of stop recursion fallback
2022-04-25 09:19:14 -04:00
'echo '$NUMBER' | wasmtime ./builtin_fib.wasm' 'echo '$NUMBER' | wasmtime ./fib_compiled_manual.wasm' \
Use hyperfine to benchmark, add builtin_fib as a comparison for how fast we could try to be
2022-04-13 00:25:53 -04:00
"scheme --script ./fib.scm $NUMBER" "scheme --script ./fib_let.scm $NUMBER" \
"python3 ./fib.py $NUMBER" "python3 ./fib_let.py $NUMBER" \
Add inlining of add and subtract, and now might be beating Python, though not by a statistically significant amount with the number of tests. Fib is still allocating 4 million times or so, which is weird, since +&- should have been the last calls to do so. Time to track that down
2022-04-20 23:47:36 -04:00
'echo '$NUMBER' | wasmtime ./rust_fib/target/wasm32-wasi/debug/rust_let.wasm' 'echo '$NUMBER' | wasmtime ./rust_fib/target/wasm32-wasi/release/rust_let.wasm' \
#'echo '$NUMBER' | wasmtime ./rust_fib/target/wasm32-wasi/debug/rust_let.wasm' 'echo '$NUMBER' | wasmtime ./rust_fib/target/wasm32-wasi/release/rust_let.wasm' \
#'echo '$NUMBER' | ./rust_fib/target/debug/rust_let' 'echo '$NUMBER' | ./rust_fib/target/release/rust_let' \
#'echo '$NUMBER' | wasmtime ./fib_interpreted.wasm' 'echo '$NUMBER' | wasmtime ./fib_interpreted_let.wasm' \
#"echo $NUMBER | java -jar ./clojure_fib/target/uberjar/clojure_fib-0.1.0-SNAPSHOT-standalone.jar" "echo $NUMBER | java -jar ./clojure_hi/target/uberjar/clojure_hi-0.1.0-SNAPSHOT-standalone.jar"
Use hyperfine to benchmark, add builtin_fib as a comparison for how fast we could try to be
2022-04-13 00:25:53 -04:00
#"./c_fib $NUMBER" "./c_fib_let $NUMBER" \
exit
Add debug_levels and turn off stack_traces by default, but save enough info about the last interaction with the top-level loop to enable re-running to problem spot with debugging on if it happens, and it works! This is the first step towards the opt/non-opt-wrap work while maintaining debugability
2022-04-09 00:45:58 -04:00
#touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib.kp && time perf record -k mono wasmtime --jitdump ./csc_out.wasm
#exit
Add fib_let test and script to run all of them. Indeed, the version with the let is worse when interpreted and the same/better when partially-evaled and compiled
2022-03-20 16:01:38 -04:00
echo "Compile Straight"
Fixed a terrible bug where turns out I used the same name for a block and a parameter in the comparison meta-function - they share the same namespace in the wasm DSL, so when I used it like a parameter in a loop it resolved to the number of scopes between the statement and the block'th parameter which had the same type and the calculation worked fine, but it overwrote the parameter I thought wasn't being used and called a function later with. Also, that seemed like that last of the really bad leaks & corruption, so re-enabling the freelist and am able to run some less-trivial benchmarks, namely (fib 30)! The compiled version is between 4.7x and 65x slower than Python, depending on if you're using wasmer, wasmtime, wasm3. Would like to try WAVM as well. A solid place to start, I think, and hopefully we'll be faster than Python once I implement a variety of dont-be-dumb optimizations (real-er malloc, static calls of statically known functions, etc)
2022-03-29 23:49:51 -04:00
#touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib.kp && time echo $NUMBER | wasm3 ./csc_out.wasm
Round allocated blocks up to the nearest 8 words, and split blocks that are >= 2x needed number of words. Now only allocates 1 wasm page for both compiled and interpreted versions at fib 30, a 269-538x improvement!
2022-03-30 21:27:01 -04:00
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib.kp && time echo $NUMBER | wasmtime ./csc_out.wasm
Add debug_levels and turn off stack_traces by default, but save enough info about the last interaction with the top-level loop to enable re-running to problem spot with debugging on if it happens, and it works! This is the first step towards the opt/non-opt-wrap work while maintaining debugability
2022-04-09 00:45:58 -04:00
#cp csc_out.wasm comp_fib_dyn.wasm
Add rust example
2022-04-11 16:07:11 -04:00
echo "Compile Straight 2"
#touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib.kp && time echo $NUMBER | wasm3 ./csc_out.wasm
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib2.kp && time echo $NUMBER | wasmtime ./csc_out.wasm
Use hyperfine to benchmark, add builtin_fib as a comparison for how fast we could try to be
2022-04-13 00:25:53 -04:00
echo "Compile Builtin"
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm builtin_fib.kp && time echo $NUMBER | wasmtime ./csc_out.wasm
echo "Interpret Straight"
#touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib.kp no_compile && time echo $NUMBER | wasm3 ./csc_out.wasm
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib.kp no_compile && time echo $NUMBER | wasmtime ./csc_out.wasm
echo "Compile Let"
#touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib_let.kp && time echo $NUMBER | wasm3 ./csc_out.wasm
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib_let.kp && time echo $NUMBER | wasmtime ./csc_out.wasm
echo "Interpret Let"
#touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib_let.kp no_compile && time echo $NUMBER | wasm3 ./csc_out.wasm
touch csc_out.wasm && rm csc_out.wasm && scheme --script ./partial_eval.scm fib_let.kp no_compile && time echo $NUMBER | wasmtime ./csc_out.wasm
Add rust example
2022-04-11 16:07:11 -04:00
echo "Python"
time python3 ./fib.py $NUMBER
echo "Rust Wasm Debug"
pushd rust_fib
cargo build --target=wasm32-wasi && time echo $NUMBER | wasmtime target/wasm32-wasi/debug/rust_let.wasm
popd
echo "Rust Wasm Release"
pushd rust_fib
cargo build --release --target=wasm32-wasi && time echo $NUMBER | wasmtime target/wasm32-wasi/release/rust_let.wasm
popd
bunch more testcases
2022-04-12 00:14:09 -04:00
echo "Rust x86 Debug"
pushd rust_fib
cargo build && time echo $NUMBER | ./target/debug/rust_let
popd
echo "Rust x86 Release"
pushd rust_fib
cargo build --release && time echo $NUMBER | ./target/release/rust_let
popd
echo "Clojure"
pushd clojure_fib
lein uberjar && time echo $NUMBER | time java -jar target/uberjar/clojure_fib-0.1.0-SNAPSHOT-standalone.jar
popd
Add debug_levels and turn off stack_traces by default, but save enough info about the last interaction with the top-level loop to enable re-running to problem spot with debugging on if it happens, and it works! This is the first step towards the opt/non-opt-wrap work while maintaining debugability
2022-04-09 00:45:58 -04:00
echo "Chez Scheme"
time scheme --script ./fib.scm $NUMBER
Add rust example
2022-04-11 16:07:11 -04:00
echo "Chez Scheme Let"
time scheme --script ./fib_let.scm $NUMBER
echo "Python"
time python3 ./fib.py $NUMBER
echo "Python Let"
time python3 ./fib_let.py $NUMBER
echo "C"
clang-11 fib.c -o fib && time ./fib $NUMBER
echo "C let"
clang-11 fib_let.c -o fib_let && time ./fib_let $NUMBER
Add fib_let test and script to run all of them. Indeed, the version with the let is worse when interpreted and the same/better when partially-evaled and compiled
2022-03-20 16:01:38 -04:00
Reference in New Issue Copy Permalink