Clean up and rearrange
This commit is contained in:
Nathan Braswell
306
doc/psudocode.txt
Normal file
306
doc/psudocode.txt
Normal file
@@ -0,0 +1,306 @@
|
||||
|
||||
Key Contributions to look out for that make this work in practical time:
|
||||
1. First class environments that:
|
||||
a. Have IDs
|
||||
b. Can either be "real", in which case it maps symbols to values,
|
||||
or "fake", in which case it maps symbols to themselves, but with the env ID as it's for-progress
|
||||
c. Chain up to an upper environment that may be fake or real
|
||||
2. AST nodes that maintain on-node:
|
||||
a. The IDs of environments that, if "real", can be used to make progress in this subtree
|
||||
b. The hashes of infinite recursive calls that were detected and stopped - if this hash isn't in the current call chain, this subtree can make progress
|
||||
c. Extra IDs of environments that are "real" but have "fake" environments in their chain - this is used to make return value checking fast O(1 or log n, depending)
|
||||
3. Combiners, both user-defined and built in (including that maintain a "wrap level" that:
|
||||
a. Is a property of this function value, *not* the function itself
|
||||
* meaning that if wrap_level > 1, you can evaluate each parameter and decrement wrap_level, even if you can't execute the call
|
||||
4. The return value of a combiner is checked for:
|
||||
a. If it is a value, in which case it is good to be returned if it doesn't contain a reference to the envID of the function it is being returned from
|
||||
b. If it is (veval something env) where env doesn't contain a reference to the envID of the function it is being returned from
|
||||
c. If it is a call to a function (func params...) and func doesn't take in a dynamic environment and params... are all good to be returned
|
||||
This makes it so that combiner calls can return partially-evaluated code - any macro-like combiner would calculate the new code and return
|
||||
(eval <constructed-code> dynamic_env), which would do what partial evaluation it could and either become a value or a call like case "b" above.
|
||||
Case "b" allows this code essentially "tagged" with the environment it should be evaluated in to be returned out of "macro-like" combiners,
|
||||
and this dovetails with the next point
|
||||
5. The (veval something env) form essentially "tags" a piece of code with the environment it should be evaluated in. At each stage where
|
||||
it is possible, the system checks for redundent constructions like these, where the env in (veval something env) is the currently active env.
|
||||
In this case, it unwraps it to just "something" and continues on - this completes the second half of the macro-like combiner evaluation where
|
||||
after being returned to the calling function the code is essentially spliced in.
|
||||
6. The compiler can emit if/else branches on the wrap_level of combiners and in each branch further compile/partial eval if appropriate, allowing
|
||||
dynamic calls to either functions or combiners with the overhead of a single branch
|
||||
|
||||
Note that points 4&5 make it so that any macro written as a combiner in "macro-style" will be expanded just like a macro would and cause no runtime overhead!
|
||||
Additionally, point 6 makes it so that functions (wrap level 1 combiners) and non-parameter-evaluating (wrap level 0) combiners can be dynamically passed around and called with very minimal overhead.
|
||||
Combine them together and you get a simpler but more flexiable semantics than macro based (pure functional) languages with little-to-no overhead.
|
||||
|
||||
Additional tricky spots to look out for:
|
||||
1. If you don't do the needed-for-progress tracking, you have exponential runtime
|
||||
2. If you aren't careful about storing analysis information on the AST node itself or memoize, a naive tree traversal of the DAG has exponential runtime
|
||||
3. Infinite recursion can hide in sneaky places, including the interply between the partial evaluator and the compiler, and careful use of multiple recursion blockers / memoization is needed to prevent all cases
|
||||
4. The invarients needed to prevent mis-evaluation are non-trivial to get right. Our invarients:
|
||||
a. All calls to user-combiners have the parameters as total values, thus not moving something that needs a particular environment underneath a different environment
|
||||
b. All return values from functions must not depend on the function's environment (there are a couple of interesting cases here, see combiner_return_ok(func_result, env_id))
|
||||
c. All array values are made up of total values
|
||||
d. Some primitive combiners don't obey "a", but they must be written with extreme care, and often partially evaluate only some of their parameters and have to keep track of which.
|
||||
|
||||
|
||||
|
||||
Everything operates on AST nodes, an ADT:
|
||||
* val - integers, strings, booleans
|
||||
* marked_array
|
||||
* marked_symbol
|
||||
* comb
|
||||
* prim_comb
|
||||
* marked_env
|
||||
|
||||
Each AST node contains a hash representing it&it's subtree.
|
||||
|
||||
fun needed_for_progress(ast_node) -> (progress_IDs, rec_stopping_hashes, extra_IDs):
|
||||
returns
|
||||
- environment IDs (stored in each AST node for it and it's children)
|
||||
that must have real values if the partial evaluation of the subtree rooted at
|
||||
this node is going to make progress partial evaluating.
|
||||
|
||||
progress_IDs is either true (meaning it will make progress no matter what), an
|
||||
intset of env IDs (the ones that will cause it to make progress), or an empty
|
||||
set, meaning it can't make forward progress no matter what
|
||||
- hashes that if you're not inside the evaluation of, it could make progress
|
||||
- extra IDs for envs it contains that don't count as forward progress IDs because the
|
||||
env does have values, but envs in it's parent chain doesn't have values.
|
||||
|
||||
The calculation for needed_for_progress is straightforward-ish, with some tricky bits at comb and array.
|
||||
|
||||
Under these definitions, we call an AST subtree a "total val" if it is either a val or it's needed-for-progress IDs is nil.
|
||||
|
||||
fun mark(x, eval_pos):
|
||||
x is env -> error
|
||||
x is combiner -> error
|
||||
x is symbol -> if x == true than MarkedVal(true)
|
||||
else if x == false than MarkedVal(false)
|
||||
else MarkedSymbol(x, needed_IDs=if eval_pos true else nil)
|
||||
x is array ->
|
||||
MarkedArray(is_val=!eval_pos, attempted=false, resume_hashes=nil,
|
||||
values = [mark(x[0], eval_pos)] + [mark(xi, false) for xi in x[1:]])
|
||||
true -> MarkedVal(x)
|
||||
|
||||
fun strip(x) -> value:
|
||||
if X is an AST node representing a value, it returns the value.
|
||||
May strip recursively in the case of an array value, etc.
|
||||
Errors on env, comb (but not prim_comb!) non value symbols or arrays
|
||||
|
||||
fun try_unval(x) -> Result<ASTNode>:
|
||||
//Removes one level of "value-ness".
|
||||
x is Array -> if !x.array_is_val Error()
|
||||
else Ok(MarkedArray(is_value=false,
|
||||
values = [try_unval(x.values[0])] + x.values[1:]))
|
||||
x is Symbol -> if !x.symbol_is_val Error()
|
||||
else Ok(MarkedSymbol(symbol=x.symbol, is_value=false))
|
||||
true -> Ok(x)
|
||||
|
||||
fun check_for_env_id_in_result(env_id, x):
|
||||
return env_id in <either progress_IDs or extra_IDs in needed_for_progress(x)>
|
||||
if either progress_IDs or extra_IDs is true, then we have a fallback, but
|
||||
that doesn't get called even on large testcases so it's either rare or impossible.
|
||||
Fallback is slow though, whereas this is just a check for set membership
|
||||
|
||||
// We only allow returning a value out of a combiner if the return value
|
||||
// doesn't reference the environment of the combiner
|
||||
fun combiner_return_ok(func_result, env_id):
|
||||
func_result isn't later -> !check_for_env_id_in_result(env_id, func_result)
|
||||
// special cases now
|
||||
(veval body {env}) => (combiner_return_ok {env})
|
||||
// The reason we don't have to check body is that this form is only creatable in ways that body was origionally a value and only need {env}
|
||||
// Either it's created by eval, in which case it's fine, or it's created by something like (eval (array veval x de) de2) and the array has checked it,
|
||||
// or it's created via literal vau invocation, in which case the body is a value.
|
||||
(func ...params) => func doesn't take dynamic env && all params are combiner_return_ok
|
||||
otherwise -> false
|
||||
|
||||
// We may end up in situations where the value/code we care about is wrapped up in
|
||||
// a redundent call to veval, namely after sucessfully returning based on combiner_return_ok above.
|
||||
// This call may prevent other optimizations though, so we should unwrap the redundent call if possible,
|
||||
// and if it causes a change we should re-partially-evaluate to make further progress if we can
|
||||
fun drop_redundent_veval(x, dynamic_env, env_stack, memostuff):
|
||||
(veval node env) if env.id == dynamic_env.id -> drop_redundent_veval(node, dynamic_env, env_stack, memostuff)
|
||||
(comb params...) if comb.wrap_level != -1 -> map drop_redundent_veval over params and if any change: partial_eval( (comb new_params...), dynamic_env, env_stack, memostuff)
|
||||
else: x
|
||||
else -> x
|
||||
|
||||
fun make_tmp_inner_env(params, de?, ue, env_id):
|
||||
...
|
||||
|
||||
|
||||
fun partial_eval_helper(x, only_head, env, env_stack, memostuff, force):
|
||||
needed, hashes, _extra = needed_for_partial_eval(x)
|
||||
if force || one of hashes is not in memostuff || needed == true || set_intersection(needed, env_stack.set_of_ids_that_are_vals) != empty_set:
|
||||
x is MarkedVal -> x
|
||||
x is MarkedEnv -> find(x.env_id == it.env_id, env_stack) ?: x
|
||||
x is MarkedComb -> if !env.is_real && !x.se.is_real // both aren't real, re-evaluation of closure creation site
|
||||
|| env.is_real && !x.se.is_real // new env real, but se isn't - the creation of the closure!
|
||||
then let inner_env = make_tmp_inner_env(x.params, x.de?, env, x.env_id)
|
||||
in MarkedComb(se=env, body=partial_eval_helper(body, false, inner_env, <add inner_env to env_stack>, memostuff, false))
|
||||
x is MarkedPrimComb -> x
|
||||
x is MarkedSymbol -> if x.is_val then x
|
||||
else env_lookup_helper(x, env)
|
||||
x is MarkedArray -> if x.is_val then x
|
||||
else let
|
||||
comb = partial_eval_helper(x.values[0], only_head=true, env, env_stack, memostuff, false)
|
||||
params = x.values[1:]
|
||||
if later_head?(comb) return MarkedArray(values=[comb]+params)
|
||||
if comb.needed_for_progress == true:
|
||||
comb = partial_eval_helper(comb, only_head=false, ...)
|
||||
|
||||
// If not -1, we always partial eval, if >0 we also unval/partial eval to do one full round of eval
|
||||
wrap_level = comb.wrap_level
|
||||
while wrap_level >= 0:
|
||||
if wrap_level >= 1:
|
||||
params = map(unval, map(\x. partial_eval_helper(x, ...), params))
|
||||
params = map(\x. partial_eval_helper(x, ...), params)
|
||||
wrap_level -= 1
|
||||
if <any of the above error, or couldn't be unvaled yet>:
|
||||
return MarkedArray(values=[comb.with_wrap_level(wrap_level)] + <params at whatever level they were sucessfully evaluated to>)
|
||||
|
||||
if comb is MarkedPrimComb:
|
||||
result = comb.impl(params)
|
||||
if result == 'LATER:
|
||||
return MarkedArray(values=[comb.with_wrap_level(wrap_level)] + params)
|
||||
else:
|
||||
return result
|
||||
|
||||
if comb.is_varadic:
|
||||
params = params[:comb.params.len-1] + [ params[comb.params.len-1:] ]
|
||||
|
||||
inner_env = MarkedEnv(id=comb.env_id, possible_de_symbol=comb.de?, possible_de=env, symbols=comb.params, values=params, upper=comb.se)
|
||||
|
||||
rec_stop_hash = combine_hash(inner_env.hash, comb.body.hash)
|
||||
if rec_stop_hash in memostuff:
|
||||
return MarkedArray(values=[comb] + params, transient_needed_env_id=true, rec_stopping_hash=rec_stop_hash)
|
||||
|
||||
memostuff.add(rec_stop_hash)
|
||||
result = partial_eval_helper(body, false, inner_env, <add inner_env to env_stack>, memostuff, false)
|
||||
memostuff.remove(rec_stop_hash)
|
||||
|
||||
if !combiner_return_ok(result, comb.env_id):
|
||||
transiently_needed = if comb.de? != nil then env.id else nil
|
||||
return MarkedArray(values=[comb] + params, transient_needed_env_id=transiently_needed, rec_stopping_hash=rec_stop_hash)
|
||||
|
||||
return drop_redundent_veval(result, env, env_stack, memostuff)
|
||||
|
||||
And then we define a root_env with PrimComb versions of all of the standard functions.
|
||||
The ones that are most interesting and interact the most with partial evaluation are
|
||||
vau eval cond
|
||||
The other key is that array only takes in values, that is an array value never hides something that isn't a total value and needs more partial-evaluation
|
||||
(this makes a lot of things simpler in other places since we can treat array values as values no matter what and know things aren't hiding in sneaky places)
|
||||
|
||||
fun needs_params_prim(...):
|
||||
...
|
||||
fun give_up_params_prim(...):
|
||||
...
|
||||
|
||||
fun veval_inner(only_head, de, env_stack, memostuff, params):
|
||||
body = params[0]
|
||||
implicent_env = len(params) != 2
|
||||
eval_env = if implicit_env { de } else { partial_eval_helper(params[1], only_head, de, env_stack, memostuff, false) }
|
||||
evaled_body = partial_eval_helper(body, only_head, eval_env, env_stack, memostuff, false)
|
||||
if implicit_env or combiner_return_ok(evaled_body, eval_env.idx):
|
||||
return drop_redundent_veval(evaled_body, de, env_stack, memostuff)
|
||||
else:
|
||||
return drop_redundent_veval(MarkedArray(values=[MarkedPrimComb('veval, wrap_level=-1, val_head_ok=true, handler=veval_inner), evaled_body, eval_env], de, env_stack, memostuff)
|
||||
|
||||
root_env = {
|
||||
eval: MarkedPrimComb('eval, wrap_level=1, val_head_ok=true, handler=lambda(only_head, de, env_stack, memostuff, params):
|
||||
let
|
||||
body = params[0]
|
||||
implicit_env = len(params) != 2
|
||||
return veval_inner(only_head, de, env_stack, memostuff, if implicit_env { [try_unval(body)] } else { [try_unval(body), params[1]] })
|
||||
)
|
||||
vapply: MarkedPrimComb('vapply, wrap_level=1, val_head_ok=true, handler=lambda(only_head, de, env_stack, memostuff, [func params env]):
|
||||
return veval_inner(only_head, de, env_stack, memostuff, [MarkedArray(values=[func]+params), env)
|
||||
)
|
||||
lapply: MarkedPrimComb('lapply, wrap_level=1, val_head_ok=true, handler=lambda(only_head, de, env_stack, memostuff, [func params env]):
|
||||
return veval_inner(only_head, de, env_stack, memostuff, [MarkedArray(values=[func.offset_wrap_level(-1)]+params), env)
|
||||
)
|
||||
vau: MarkedPrimComb('vau, wrap_level=0, val_head_ok=true, handler=lambda(only_head, de, env_stack, memostuff, params):
|
||||
let
|
||||
de? = if len(params) == 3 { params[0].symbol_value } else { nil }
|
||||
params = map(lambda(x): s.symbol_value, if de? { params[1] } else { params[0] })
|
||||
varadic = '& in params
|
||||
params.remove('&)
|
||||
implicit_env = len(params) != 2
|
||||
body = try_unval(if de? { params[2] } else { params[1] })
|
||||
env_id = <new_id>
|
||||
if !only_head:
|
||||
inner_env = make_tmp_inner_env(params, de?, upper=de, id=env_id)
|
||||
body = partial_eval_helper(body, false, inner_env, <add inner_env to env_stack>, memostuff, false)
|
||||
return MarkedComb(wrap_level=0, id=new_id, de?=de?, static_env=de, variadic=varadic, params=params, body=body)
|
||||
)
|
||||
wrap: ...<returns new MarkedPrimComb/MarkedComb with incremented wrap_level>...
|
||||
unwrap: ...<returns new MarkedPrimComb/MarkedComb with decremented wrap_level>...
|
||||
cond: ...
|
||||
...Oddly tricky - is wrap_level 0, but...
|
||||
... 1. unvals & partially evaluates starting from the first condition
|
||||
... 2. if this condition is true, return the unvald & partially evaluated corresponding arm
|
||||
... 3. if this condition is false, drop the arm and return to 1
|
||||
... 4. In this case, we have an unknown between true & false
|
||||
... 5. check to see if combine_hash(x.hash, env.hash) is in memostuff (prevent infinite recursion blocked on a cond guard!)
|
||||
... 6. if the hash was in memostuff, return MarkedArray(later_hash=the_hash,
|
||||
... values=[MarkedPrimComb('vcond,wraplevel=-1,...)] + map(unval, <remaining preds/arms>))
|
||||
... 7. else new_preds_arms = map(partial_eval..., map(unval, <remaining preds/arms>))
|
||||
... <TODO: 8. remove arms/preds now guarenteed to be false, remove all arms/preds after first true>
|
||||
... 9. return MarkedArray(values=[MarkedPrimComb('vcond,wraplevel=-1,...)] + new_preds)
|
||||
...
|
||||
...The vcond is like cond but doesn't do any unvaling (as it's already been done) (and wrap_level is set to -1 so the function call machinery doesn't touch the params either)
|
||||
...
|
||||
symbol?: needs_params_prim(symbol?)
|
||||
int?: needs_params_prim(int?)
|
||||
string?: needs_params_prim(string?)
|
||||
combiner?: ...
|
||||
env?: ...
|
||||
nil?: needs_params_prim(nil?)
|
||||
bool?: needs_params_prim(bool?)
|
||||
str-to-symbol: needs_params_prim(str-to-symbol)
|
||||
get-text: needs_params_prim(get-text)
|
||||
array?: ...
|
||||
array: ...
|
||||
len: ...
|
||||
idx: ...
|
||||
slice: ...
|
||||
concat: ...
|
||||
+: needs_params_prim(+)
|
||||
-: needs_params_prim(-)
|
||||
*: needs_params_prim(*)
|
||||
/: needs_params_prim(/)
|
||||
%: needs_params_prim(%)
|
||||
band: needs_params_prim(band)
|
||||
bor: needs_params_prim(bor)
|
||||
bnot: needs_params_prim(bnot)
|
||||
bxor: needs_params_prim(bxor)
|
||||
<<: needs_params_prim(<<)
|
||||
>>: needs_params_prim(>>)
|
||||
=: needs_params_prim(=)
|
||||
!=: needs_params_prim(!=)
|
||||
<: needs_params_prim(<)
|
||||
<=: needs_params_prim(<=)
|
||||
>: needs_params_prim(>)
|
||||
>=: needs_params_prim(>=)
|
||||
str: needs_params_prim(true_str)
|
||||
log: give_up_params_prim(log)
|
||||
error: give_up_params_prim(error)
|
||||
read-string: needs_params_prim(read-string)
|
||||
empty_env: MarkedEnv()
|
||||
}
|
||||
|
||||
fun compile(...):
|
||||
...
|
||||
... tagged words, etc
|
||||
... eval
|
||||
... vau / vau helper closure
|
||||
...
|
||||
Note that when it's compiling a call, it compiles an if/else chain on the wrap level of the combiner being called.
|
||||
in the 0 branch, it emits the parameters as constant data
|
||||
in the 1 branch, it unval's and partial evals all of the parameters before compiling them.
|
||||
- note that this must be robust to partial-eval errors, as this branch might not ever happen at runtime and be nonsense code!
|
||||
- if the partial evaluation errors, it emits a value that will cause an error at runtime into the compiled code
|
||||
in the > 1 branch, it errors
|
||||
...
|
||||
...
|
||||
Must be careful about infiniate recursion, including tricky cases that infinitly ping back and forth between
|
||||
partial eval and compile even though both have individual internal recursion checks
|
||||
...
|
||||
Reference in New Issue
Block a user