hi everybody my name is Alan diapered
and I'll be talking today about
JavaScript assisted common lisp or as I
like to call it jackal the jackal is a
new Common Lisp implementation that
targets the JavaScript platform my
agenda consists of three major topics
first of all I'd like to set the scene
for you to give you a little context and
background that could help you
understand jackal and determine whether
or not it's something that's applicable
to what you do I find that projects are
much easier to understand if there's a
if if you have a little background about
why that project came to be and what
motivated it what experience experience
has led to its creation and in this part
of my talk all hope to illuminate that
for you I'll also talk about the
JavaScript platform and why I'm
interested in it what it can do
and what it's not what it can't do and
set the stage technically for jackal
then we'll get into jackal itself its
high-level organization the features
that I think are interesting and
important and the parts of it that are
maybe speculative finally I'll do a
little demonstration and we can see
jackal in the browser running and I'll
also show jackals repple client facility
which is unkind of a novel thing about
jackal because it involves this
asynchronous reader that I'll talk about
my wrote to jackal began I think around
1994 I was learning basic at the time I
think quick basic and my dad who was a
taught me programming has been a big
inspiration for a lot of things that I
do after I learned about arrays he said
Allan arrays are really cool aren't they
and I said yeah yeah very cool and he
said to me you know there's a whole
language oriented around the idea of
arrays and I distinctly remember
thinking wow that sounds so dumb
and I'm sure he laughed a little scoffed
and brushed aside my absolutely ignorant
comment and I went on to program and a
few other kinds of languages I returned
to Lisp in a big way with in 2009 when I
learned about closure and I started to
use closure professionally and a few
years after that I was involved in an
effort to create something called
closure script which was a variant of
closure that compiled to JavaScript
instead of to the JVM but still
emphasizing functional programming
closure script was really eye-opening to
me in a couple regards first of all it
was the first time I had been involved
in any kind of compiler development and
I got to see how that was done
that was really interesting and maybe my
abiding memory from that experience is
just that wow it's you could you know
even lowly application developers like
myself can benefit from making the right
kind of compiler to fit the right kind
of problem it's a great tool to be able
to make and to apply the other thing
about it was that closure script from
the beginning was really oriented toward
industrial application and performance
both in terms of code size and in terms
of speed are really important in the
JavaScript world because of various
limitations surrounding JavaScript
including the fact that when people run
your program they need to download the
whole program every time usually and the
designer of closure script rich Hickey
he made a very I think yeah he had a
very key insight related to code size
and then code improvement which was it's
possible to lean on third-party tools
like Google closure compiler to do that
part of the compilation pipeline and the
advantage of that is you can keep your
main compiler quite simple and easy to
work on
so fast-forward a few years I did a
bunch of industrial closure script
development I used it at work and
successfully and continue to do closure
around 2016 sort of randomly just got
into common list but we had been always
kind of a blind spot for me in the list
world and I knew rich Hickey had been
influenced by common lisp before he made
closure and I had brushes with it
through closure but I'd never really sat
down with it myself and the more time I
spent with it the more I grew to really
really liked it it's I think in a lot of
respects lower level than closure but I
think that that lower level aspect
defers more capability to the programmer
there's more room to make the language
of the problem as I've heard it called
so as my Lisp fanaticism grew I had this
idea to make jackal but there actually a
few other ideas that that led to jackal
that happen kind of concurrently at one
point I had the idea to make my own lisp
dialect called beta Lisp that was going
to be an experimental vehicle to play
with this idea that macros receive
arguments macro expanded by DeVault and
so the beta and beta List comes from
beta reduction lambda calculus I had
this idea that that would you know
dramatically clean up the lisp
implementation but didn't really work
out that way there are other problems
associated with that and I didn't get
too far with it but I did stumble on
this idea of an asynchronous reader
which I'll talk about again soon in the
talk at one point I picked up a copy of
Common Lisp the language second edition
by guy Steele and I also read through a
bunch of the early Common Lisp design
discussion emails and this su - AI DARPA
mailing list thing and that got me
interested in or I think gave me an
appreciation for the amount of design
work that went into Common Lisp which
was really refreshing to me because I
had just failed to design a language and
I felt better about failing to do so
when I realized how much how difficult
that really was to do and I realized
that implementing a spec
he's gonna would be a lot easier than
designing a language like orders of
magnitude and at least one reason is if
you're handed a spec you can work on it
in pretty small increments and make
really gratifying progress over a long
period of time and that progress be
continuous but if you're working on a
design of a language that's that that's
something that requires a lot more
effort so I just started to play with
this idea of implementing a common lisp
and I knew I knew common list was huge
and that it would probably never be done
but I just couldn't help myself and as I
got going I realized I had this
opportunity to maybe maybe achieve some
things with a new Common Lisp first I
knew that the browser platform suggest a
lot of language level ideas to solving
various problems and I thought you know
if I'm gonna be making languages to
target JavaScript what better way to
start than with common lisp as a
substrate because it's it's a known
quantity and it's a proven substrate and
I also realized I had an opportunity to
pursue this idea of an asynchronous
reader because I had a suspicion that
with a annecy asynchronous reader a new
level of residential development could
become possible which is to say that the
development time and the runtime
environments could be integrated in a
way that they haven't yet been with any
of the Lisp implementations that I'd
experimented with and plus you can't go
wrong if you have you have a nice Common
Lisp to work with and I thought you know
if I'm going to be doing browser stuff
for the rest of my career be nice to
have a common lisp that I can I can use
so that I can write Common Lisp so the
JavaScript platform which I'm interested
in compiling to in which jackal is
intimately related in a nutshell it
makes the browser programmable so prior
to the advent of JavaScript the web
browsers were effectively static
document viewers with some styling
capability javascript makes them
full-blown programming environments
it's interesting and important and
valuable in an industrial setting
because it makes people feel like
webpages are faster they're not
necessarily actually doing work faster
but they're more responsive modern
JavaScript runtimes are also quite
efficient and fast they have world-class
garbage collectors now in Firefox and
Chrome and world-class tracing Jets
really incredible technology the
javascript is quite a fast language even
in commodity web browsers there's a
document object model a model extra
abstraction in JavaScript and this is
how one presents user interfaces and
collects input from the browser the
JavaScript platform is heavily heavily
restricted and it has severe limitations
and it has those limitations because
it's fundamentally for running untrusted
code people have to believe that if they
have JavaScript on and their web browser
and they go to a website that the
website will not have access to their
file their personal files and will not
be able to crash their computer so
that's probably the major reason that
javascript does not have multiple
threads multiple any kind of concurrent
execution model it's single process and
it has an event loop which is to say
that inputs can happen in JavaScript but
inputs are in queued on an event queue
and that event queue can can can only be
consumed by one process which is the
single JavaScript process
so jackal brings to bear in the
JavaScript arena a few a few different
tools the first set of tools is related
to inter operation with JavaScript and I
mentioned some a few special forms here
that allow one to call JavaScript
functions access JavaScript object
fields and to generate obvious arbitrary
JavaScript code there are other aspects
of this that aren't on the slide like
you know the jackal numeric type the fix
num is a JavaScript floating-point
number so the relationship there is very
tight and it's actually that tightness
is an effort towards efficiency since I
don't need to ship my own
implementations of things like fixed
noms or flown ohms but that level of
integration is also primarily what will
preclude jackal from becoming any kind
of you know a compliant conforming
implementation javascript is just too
too restrictive for that's ever be
possible another the second major aspect
of jackal is that it needs to be able to
make small and efficient code as I
mentioned earlier I want to use this at
work someday and needs to be competitive
with whatever language to your people in
that in my field are using and I learned
from the closure script experience that
I don't necessarily need to be the
person who writes those optimizations
because I just need to generate
JavaScript that other JavaScript
optimizers can understand and this is
great as you know an individual
contributors this is my personal project
this means that I can focus on other
parts that of the codebase and defer
optimizations to third-party tools which
are excellent the third thing that
jackal focuses on which is the most
speculative but also maybe the most
interesting is that I have this idea
that there is advantage to doing what
has been called residential development
and residential development is a
development workflow and sort of ethos
characterized by this idea of closely
integrating the runtime
and the development environments so the
same tools that you use to build your
application might be tools that you
would use in the application itself and
the line between application and tooling
is very fuzzy interesting consequences
of this include the idea that
development tools can inhabit the
application through to deployment and
even after small talk and Lisp both
historically supported this kind of
development although an IDE like a lisp
machine or a small talk environment is
not really in my personal scope that's
just a little more work that I'm willing
to take on and I'm not particularly
interested in doing that but I think
there might be advantaged in not
precluding that because whatever tools I
develop would converge on an integrated
development environment and I just
wouldn't want to make that kind of thing
impossible and in the meantime I
personally use a Emacs with a network
repple which I'll demonstrate so I
mentioned that the a asynchronous reader
was kind of a banner feature in
Jaqueline and the reason for that is
kind of a linchpin of the residential
development environment workflow
experiment the fundamental problem here
is Java scripts model of input input can
only be consumed asynchronously via
callback so if one were to say consume
characters in JavaScript you wouldn't be
able to read new characters from some
input source until your code is done
running because there's a single process
code can only because objects would only
be consumed from the event loop of the
event queue once the stack is empty so
there's no blocking which means that the
straightforward traditional approach to
a lisp reader is not really effective
because you can make a lisp reader in
the traditional style but it will block
everything else in the browser tab and
your program could be deadlocked easily
so in jackal what we do is we have an
asynchronous reader abstraction which
it's a stateful object you instantiate
one of these asynchronous readers and it
has a method for pushing characters into
it and every time you call the method
and push characters internally
incrementally it parses with an
incremental it's an online incremental
parser and when it's accumulated a data
structure with it that's a list or a
symbol or whatever it will invoke a
callback with that data structure so
next up is a demonstration and I'll show
interaction in the browser and what you
other sort of what it looks like to run
Jackel in the browser and I'll show the
integration with JavaScript and we'll
show the asynchronous reader and then
I'll show what the development
experience currently looks like using
this Jackal ripple client connecting to
a remote browser and evaluating okay we
are back now with a demonstration so
over on the left hand side here we've
got a terminal open and I'm running a
local web server on port 8000 from the
Jackal distribution directory and on the
right hand side we're looking at the
Jackal tests which are just basic tests
of Jackel woefully incomplete but the
important thing about the browser on the
right is that Jackal is loaded into it
and we can actually interact with Jackel
through the browser's built-in
development tools and I'll show you how
that's done here I'll make this font a
little bigger so I happen to know that
there is the moral equivalent of an
input stream set up in this browser
ready to receive characters from us
now of course this console here is a
JavaScript console but we can write
JavaScript code that pushes characters
into the compilers reader or with the
Rebels reader and I think it's called
ripple input stream so we can do Ruppel
input stream
right each which will take a sequence of
characters and we can send it just the
number one two three cool the killer
feature of the jackal ripple is that
it's incremental so when we sent the
characters one two three with a terminal
space it read the token but because this
is an incremental parser we can send it
characters one at a time and it will
digest them and only read compile and
evaluate when an entire token or an
entire data has been accumulated a list
is of course the best demonstration of
this so make a list here with just one
and two I'd better quote it and let's
send in three see the effect now we sent
in a list which you can see gets
evaluated and compiled or compiled and
evaluated as a cons type object and we
can inspect that object using the
JavaScript object inspector so you can
see it has two slots car and cutter
cutter is a cons with 23 and then the
car of that excuse me the cutter of that
second cons is null this is a proper
list we can also read symbols so let me
read a symbol and the symbol can be our
entry point into the package system oh
of course it's not done reading that
symbol because I have to send it some
white space but when it did you can see
we get a symbol back and you can see
that there's a lot more going on with
symbols than with cons as' plus here is
refers to the plus from the Common Lisp
user package so you can see that we're
in an environment where the CL package
has not been referred
you can see some metadata here that's
interesting to the the runtime like
whether it's constant a macro special
variable and yeah you can see the stack
field here this has to do with
supporting dynamic binding okay so that
actually wasn't as interesting as I had
hoped so let's use a different runtime
facility to look at the all of the
packages that are loaded so I think I
can do packages so that's a JavaScript
global variable called all caps packages
and you can see that it contains many
entries including jackal jeaious Common
Lisp and then some aliases that keyword
package of course let's drill into the
common list package well you can see the
list of exports and all of the symbols
that are interned in there so if we can
find a symbol that's a function like not
okay we can see that's a value the list
symbol is the value and it has an F
value so this is a symbol with a
function slot of or an F definition of a
particular function and that function is
a JavaScript function and so we could
actually right click here to store as
global variable and now in the
JavaScript environment we have a handle
to this function and you can see it's a
compiled function and let's call it so
if we call it with null it returns true
if we call it with true returns null so
all of this is to demonstrate that
Jackal functions are JavaScript
functions you can see here that the
Jackal function does an argument length
checking which javascript does not do by
default okay
enough tinkering in the browser let's do
some stuff from Emacs because it's not a
real Lisp until doing stuff for me backs
right
so flipping over to Emax here these are
the jackal tests actually all this this
test results screen here is run as a
consequence of this code running I just
have this open there just to show some
jackal code but we want to start a
ripple and the way we can do that is
with the jackal tool that ships with the
jackal distribution currently so this
browser on the right hand side is I
started it with the proper command line
flag to turn on remote debugging and the
jackal repple tool leverages the fact
that the browser is effectively a server
and it connects to that server in order
to send it data characters so just start
this up I do in Emacs I can do ctrl you
meta X run - Lisp and instead of the
list command I'm going to give it a
jackal command and this is actually
hosed in some way oh right because we
have a dev tool session over here we
can't start a new one over any max so
I'm gonna reboot this then I'm gonna use
a shortcut I have an Emacs called jackal
repple this function eat list function
okay now over in the Emacs ID we're
connected up to the browser on the right
hand side so I can do stuff but you're
not really seeing it take effect because
evaluating the number one two three
doesn't do anything in the browser but
we can use the jackal special forms to
do things like call the alert function
to produce a box and on the browser so
let me do well first let me go down to
the jackal tests file and I'm going to
evaluate this to enable some of the
reader macros for Jo syntax so we're
interested in the alert function and I'm
going to reference it here from jackal
you can see the repple doesn't pretty
print stuff and it's just basically
spewing the javascript textual
representation of things but that @pipe
reader macro is a way for us to access
variables so we can do we can use the
the well has a couple ways we can do
this we can use jackal call and pass it
a topic or a function the alert and then
let's pass it a JavaScript string boom
cool we can use a little more sugar to
do something like that oh I didn't like
it oh I think I know how something might
be broken there anyway more work for me
to do but this concludes the demo and I
thank you very much for your time and I
hope you learned something from my
presentation I look forward to answering
your questions thank you very much