If you don’t know it already, Crystal is a programming language with syntax and semantics similar to Ruby, except that it is not interpreted, it compiles programs to native code.
So we chose to implement a compiler for Crystal in Ruby. Why?
- Ruby is an awesome language with an elegant syntax, suitable for very fast prototyping.
- One day we could write the compiler in Crystal, and porting the code should be relatively easy if the syntax and semantic are similar to Ruby.
We tried many times to implement the compiler in Crystal. But we always had problems with it.
Compile times were too big
This, we thought at first, is because Ruby is sometimes slow.
However, in the beginning the language was pure, very, very similar to Ruby where you never have to specify types. Compilation times were growing exponentially relative to the code size and to the amount of Array instantiations. Trying to compile just part of the compiler started to take minutes. We decided that this was unacceptable.
So we made a small sacrifice: you sometimes have to specify the types of arrays, hashes, and other generic types.
a = [] # OK for Ruby, but not for Crystal
b = [] of Int32 # OK for Crystal
c = [1, 2, 3] # OK for Ruby and for Crystal
c << 4 # OK for Ruby and for Crystal
c << "hello" # OK for Ruby, error for Crystal (c is Array(Int32))
d = [1, 2, 3] of Int32 | String
d << "hello" # OK for CrystalWith this little change compile times were much better, growing lineraly relative to the code size.
But still we couldn’t finish the new compiler.
Lack of features
In the compiler written in Ruby we used Array, Hash and Set. We accesed the filesystem and we used bindings to LLVM. Unless we had those same features in our language and standard library, we would never be able to implement a compiler in Crystal.
So we added a lot of funcionality to the standard library. We added bindings to C. We have C structs and unions. We have function pointers. And all of these are specified in Crystal, no need to write those in another language.
But still…
Bugs
The compiler was not perfect. It had (and still has) bugs. Everything worked fine for small examples and for our tests, but writing a compiler in Crystal really started to exercise the compiler in Ruby and to reveal lots of bugs and missing features.
So we fixed most of these bugs and added the missing features.
But all of this always meant:
Lagging behind the compiler
Every bug fix or new feature we introduced in the compiler written in Ruby had to be ported to the compiler written in Crystal (which still didn’t successfully compile). And every now and then we were tempted to add new features to the language, and we did, so we started to lag more and more.
So one day we decided to do a feature freeze (and also a bug freeze, unless we could workaround the problem).
It was a long path, but also a very interesting and enlightening one:
- Porting the code from Ruby to Crystal is very easy and most of the time needs few modifications.
- The ported code behaved exactly the same as in Ruby. We are still amazed that we could capture Ruby's syntax and semantic so well.
- The ported code revealed bugs in the compiler written in Ruby, which means that, in theory, Crystal helps you have more robust and correct code.
And today, Thursday, we finally did it. We managed to write a compiler for Crystal written in Crystal itself. Yay! The new compiler can compile itself successfully, and this new compiler can compile itself, and the resulting binary is exactly the same as the old one. It can also compile its specs, and they all pass.
The future for Crystal looks bright:
It is fast(er)
The compiler written in Ruby takes about 20 seconds to infer the types of the compiler. The compiler written in Crystal takes about 2.8 seconds to do the same.
Remember: we are talking about global type inference here. 2.8 seconds to do semantic analysis for a compiler. Not bad at all!
And we still have many optimizations to apply, both to the code generated by the compiler, to the code present in the compiler and standard library (for instance, Hash has a very naive implementation).
We don’t depend on Ruby anymore
Because we now have a compiler written in Crystal, we can compile new versions of the compiler just using Crystal. Good bye, Ruby. It was a pleasure to have you on our team, but, well, you were a bit slow for our needs. Yes, yes, we like a lot of things about you, but this is a compiler we are talking about here. You can’t just have programmers wait minutes and minutes to compile their programs. Oh, maybe we are being a bit harsh on you. You know what? Don’t go away. Come back. You can still help us develop fantastic web application front ends with that friend of yours, Rails. We mean it, seriously. You shine in this. We are not so sure about our backends, though. Erlang and Go really rock for this. Too bad their syntax (and semantic) is not as nice as yours. Will there someday be a language like you, but fast like native code? Probably not. But how about a similar language, where you have to make some small sacrifices if you come from Ruby? We really hope so.
Roadmap
Now we have to fix the remaining bugs in the compiler. We don’t like buggy software so we won’t just continue adding features to the language unless we make it rock solid.
Then we can start thinking about concurrency, better macros, better funcion pointers, structs for real, named arguments, tuples, fibers, a debugger…
If you don’t know it already, Crystal is a programming language with syntax and semantics similar to Ruby, except that it is not interpreted, it compiles programs to native code.
So we chose to implement a compiler for Crystal in Ruby. Why?
We tried many times to implement the compiler in Crystal. But we always had problems with it.
Compile times were too big
This, we thought at first, is because Ruby is sometimes slow.
However, in the beginning the language was pure, very, very similar to Ruby where you never have to specify types. Compilation times were growing exponentially relative to the code size and to the amount of Array instantiations. Trying to compile just part of the compiler started to take minutes. We decided that this was unacceptable.
So we made a small sacrifice: you sometimes have to specify the types of arrays, hashes, and other generic types.
With this little change compile times were much better, growing lineraly relative to the code size.
But still we couldn’t finish the new compiler.
Lack of features
In the compiler written in Ruby we used Array, Hash and Set. We accesed the filesystem and we used bindings to LLVM. Unless we had those same features in our language and standard library, we would never be able to implement a compiler in Crystal.
So we added a lot of funcionality to the standard library. We added bindings to C. We have C structs and unions. We have function pointers. And all of these are specified in Crystal, no need to write those in another language.
But still…
Bugs
The compiler was not perfect. It had (and still has) bugs. Everything worked fine for small examples and for our tests, but writing a compiler in Crystal really started to exercise the compiler in Ruby and to reveal lots of bugs and missing features.
So we fixed most of these bugs and added the missing features.
But all of this always meant:
Lagging behind the compiler
Every bug fix or new feature we introduced in the compiler written in Ruby had to be ported to the compiler written in Crystal (which still didn’t successfully compile). And every now and then we were tempted to add new features to the language, and we did, so we started to lag more and more.
So one day we decided to do a feature freeze (and also a bug freeze, unless we could workaround the problem).
It was a long path, but also a very interesting and enlightening one:
And today, Thursday, we finally did it. We managed to write a compiler for Crystal written in Crystal itself. Yay! The new compiler can compile itself successfully, and this new compiler can compile itself, and the resulting binary is exactly the same as the old one. It can also compile its specs, and they all pass.
The future for Crystal looks bright:
It is fast(er)
The compiler written in Ruby takes about 20 seconds to infer the types of the compiler. The compiler written in Crystal takes about 2.8 seconds to do the same.
Remember: we are talking about global type inference here. 2.8 seconds to do semantic analysis for a compiler. Not bad at all!
And we still have many optimizations to apply, both to the code generated by the compiler, to the code present in the compiler and standard library (for instance, Hash has a very naive implementation).
We don’t depend on Ruby anymore
Because we now have a compiler written in Crystal, we can compile new versions of the compiler just using Crystal. Good bye, Ruby. It was a pleasure to have you on our team, but, well, you were a bit slow for our needs. Yes, yes, we like a lot of things about you, but this is a compiler we are talking about here. You can’t just have programmers wait minutes and minutes to compile their programs. Oh, maybe we are being a bit harsh on you. You know what? Don’t go away. Come back. You can still help us develop fantastic web application front ends with that friend of yours, Rails. We mean it, seriously. You shine in this. We are not so sure about our backends, though. Erlang and Go really rock for this. Too bad their syntax (and semantic) is not as nice as yours. Will there someday be a language like you, but fast like native code? Probably not. But how about a similar language, where you have to make some small sacrifices if you come from Ruby? We really hope so.
Roadmap
Now we have to fix the remaining bugs in the compiler. We don’t like buggy software so we won’t just continue adding features to the language unless we make it rock solid.
Then we can start thinking about concurrency, better macros, better funcion pointers, structs for real, named arguments, tuples, fibers, a debugger…