Use above macros to create richer language and embed entire other programming languages (syntax, semantics, and type system) for flawless interop/FFI (C, Go, Lua, JS, etc)
File is interpreted, and then if "main" exists it is compiled, spidering backwards to referenced functions and data (Allows interpreted code to do metaprogramming, dependency resolution, generate code, etc, which is then compiled)
Regionalized Value State Dependence Graph as backend-IR, enabling simpler implementations of powerful optimizations (RSVDG paper) so that embedded languages have good preformance when compiled with little code
About:
Currently, I am bootstrapping this new core Lisp out of my prior compiler for my programming language, Kraken. I have implemented the first version of the FUN-GLL algorithm and have working context-free reader macros. I'll have enough to self-host this core soon, and will then use the more efficent core Lisp implementation to implement the Type Systems as Macros paper and add a type system to the new language.
The general flow is that the input file is executed with the core Lisp interpreter, and if there is a "main" symbol defined the compiler emits C for that function & all other functions & data that it references. In this way the language supports very powerful meta-programming at compile time, including adding syntax to the language, arbitrary computation, and importing other files, and then compiles into a static executable. The current compiling backend emits C.
Below are a few examples of using the live grammer modification / context-free reader macros to implement basic methods as well as embed the BF language into the core Lisp. The core Lisp implementation has been compiled to WebAssembly and should be able to run in your browser. Feel free to make edits and play around below.
Note that the current implementation is inefficent, and sometimes has problems running in phone web browsers.
Runnable Example Code:
; Of course
(println "Hello World")
; Just print 3
(println "Math works:" (+ 1 2))
Output:
Method Example:
Let's use our meta system (attaching objects to other objects) to implement basic objects/methods.
We will attach a vector of alternating symbols / functions (to make this example simple, since maps aren't built in) to our data as the meta, then look up methods on it when we preform a call. The add_grammer_rule function modifies the grammer/parser currently being used to parse the file and operates as a super-powerful reader macro. We use it in this code to add a rule that transforms
a.b(c, d)
into
(method-call a 'b c d)
where method-call is the function that looks up the symbol 'b on the meta object attached to a and calls it with the rest of the parameters.
; First quick lookup function, since maps are not built in
(def! get-value-helper (fn* (dict key idx) (if (>= idx (count dict))
nil
(if (= key (nth dict idx))
(nth dict (+ idx 1))
(get-value-helper dict key (+ idx 2))))))
(def! get-value (fn* (dict key) (get-value-helper dict key 0)))
; Our actual method call function
(def! method-call (fn* (object method & arguments) (let* (method_fn (get-value (meta object) method))
(if (= method_fn nil)
(println "no method " method)
(apply method_fn object arguments)))))
; Some nice syntatic sugar for method calls
(add_grammer_rule 'form ['form "\\." 'atom 'optional_WS "\\(" 'optional_WS 'space_forms 'optional_WS "\\)"]
(fn* (o _ m _ _ _ p _ _) `(method-call ~o '~m ,p)))
; Ok, let's create our object by hand for this example
(def! actual_obj (with-meta [0] [
'inc (fn* (o) (set-nth! o 0 (+ (nth o 0) 1)))
'dec (fn* (o) (set-nth! o 0 (- (nth o 0) 1)))
'set (fn* (o n) (set-nth! o 0 n))
'get (fn* (o) (nth o 0))
]))
(do
; Use our new sugar
actual_obj.set(1337)
actual_obj.inc()
(println "get: " actual_obj.get())
actual_obj.dec()
(println "get: " actual_obj.get())
; Use methods directly
(method-call actual_obj 'set 654)
(method-call actual_obj 'inc)
(println "get: " (method-call actual_obj 'get))
(method-call actual_obj 'dec)
(method-call actual_obj 'dec)
(println "get: " (method-call actual_obj 'get))
nil)
Output:
More Complicated Example: BF as an embedded language
; We don't have atoms built in, mutable vectors
; are our base building block. In order to make the
; following BF implementation nice, let's add atoms!
; They will be implmented as length 1 vectors with nice syntax for deref
(def! make-atom (fn* (x) [x]))
(def! set-atom! (fn* (x y) (set-nth! x 0 y)))
(def! get-atom (fn* (x) (nth x 0)))
(add_grammer_rule 'form ["@" 'form] (fn* (_ x) `(get-atom ~x)))
; Now begin by defining our BF syntax & semantics
; Define our tokens as BF atoms
(add_grammer_rule 'bfs_atom ["<"] (fn* (_) '(set-atom! cursor (- @cursor 1))))
(add_grammer_rule 'bfs_atom [">"] (fn* (_) '(set-atom! cursor (+ @cursor 1))))
(add_grammer_rule 'bfs_atom ["\\+"] (fn* (_) '(set-nth! tape @cursor (+ (nth tape @cursor) 1))))
(add_grammer_rule 'bfs_atom ["-"] (fn* (_) '(set-nth! tape @cursor (- (nth tape @cursor) 1))))
(add_grammer_rule 'bfs_atom [","] (fn* (_) '(let* (value (nth input @inptr))
(do (set-atom! inptr (+ 1 @inptr))
(set-nth! tape @cursor value)))))
(add_grammer_rule 'bfs_atom ["."] (fn* (_) '(set-atom! output (cons (nth tape @cursor) @output))))
; Define strings of BF atoms
(add_grammer_rule 'bfs ['bfs_atom *] (fn* (x) x))
; Add loop as an atom
; (note that closure cannot yet close over itself by value, so we pass it in)
(add_grammer_rule 'bfs_atom ["\\[" 'bfs "]"] (fn* (_ x _)
`(let* (f (fn* (f)
(if (= 0 (nth tape @cursor))
nil
(do ,x (f f)))))
(f f))))
; For now, stick BFS rule inside an unambigious BFS block
; Also add setup code
(add_grammer_rule 'form ["bf" 'optional_WS "{" 'optional_WS 'bfs 'optional_WS "}"]
(fn* (_ _ _ _ x _ _)
`(fn* (input)
(let* (
tape (vector 0 0 0 0 0)
cursor (make-atom 0)
inptr (make-atom 0)
output (make-atom (vector))
)
(do (println "beginning bfs") ,x (nth output 0))))))
; Let's try it out! This BF program prints the input 3 times
(println (bf { ,>+++[<.>-] } [1337]))
; we can also have it compile into our main program (if this wasn't the web version)
(def! main (fn* () (do (println "BF: " (bf { ,>+++[<.>-] } [1337])) 0)))
Output:
Performance Benchmarks
Performance is quite poor, as almost no work has gone into it as of yet.
We are currently focusing on the FUN-GLL macros and creating a more fully-featured language on top of the core Lisp using them. We will focus more on performance with the implemenation of the functional persistant datastructures and the self-hosting rewrite, and performance will be the main focus of the RVSDG IR part of the project.
Even so, it is worth keeping a rough estimate of performance in mind. For this, we have compiled a very basic benchmark below, with more benchmark programs (sorting, etc) to be included as the language gets developed:
Core Lisp Interpreter
Core Lisp Compiled to C
Hand-written C
Fibonacci(27)
51.505s
0.007s
0.002s
Next Steps
Implement simple garbage collector for compiled code (currently C)
Implement persistant functional data structures
Hash Array-Mapped Trie (HAMT) / Relaxed Radix Balance Tree (RRB-Tree)
Hash Map based on the above
Hash Set based on the above
Prototype Type Systems as Macros, may require macro system rewrite/upgrade
Sketch out Kraken language on top of core Lisp, includes basic Hindley-Milner type system implemented with Macros and above data structures
Re-self-host using functional approach in above Kraken language
Use Type System Macros to implement automatic transiant creation on HAMT/RBB-Tree as an optimization
Implement RVSDG IR and develop best bang-for-buck optimizations using it