Exciting times! I’m moving from Newcastle University to Jane Street London to work on the OCaml build system Dune. I’ve received a lot of questions about this move already, so I decided to write down some answers in a blog post.
Tag Archives: build
Stroll: an experimental build system
I’d like to share an experiment in developing a language-agnostic build system that does not require the user to specify dependencies between individual build tasks. By “language-agnostic” I mean the user does not need to learn a new language or a special file format for describing build tasks — existing scripts or executable files can be used as build tasks directly without modification.
I call this build system Stroll because its build algorithm reminds me of strolling through a park where you’ve never been before, trying to figure out an optimal path to your target destination, and likely going in circles occasionally until you’ve built up a complete mental map of the park.
You should try Hadrian
This is an announcement for GHC developers:
You should try to use Hadrian as the GHC build system, because it will (hopefully!) become the default around GHC 8.8.
Build Systems à la Carte
In a recent blog post, I shared a preliminary version of the paper on build systems that Neil Mitchell, Simon Peyton Jones and I submitted to the ICFP 2018 conference. The paper was accepted and yesterday, after months of revisions and polishing, we’ve finally completed this work. The paper and associated executable models are openly available; here is a direct link to the PDF.
Build systems, such as classic Make, are big, complicated, and used by every software developer on the planet. But they are a sadly unloved part of the software ecosystem, very much a means to an end, and seldom the focus of attention. Rarely do people ask questions like “What does it mean for my build system to be correct?” or “What are the trade-offs between different approaches?”. For years Make dominated, but more recently the challenges of scale have driven large software firms like Microsoft, Facebook and Google to develop their own build systems, exploring new points in the design space. In this paper we offer a general framework in which to understand and compare build systems, in a way that is both abstract (omitting incidental detail) and yet precise (implemented as Haskell code).
Hadrian is on the way
Hadrian, a new build system for GHC that we have been working on for the past three years, has finally been merged into the GHC tree (update: we have temporally switched to a submodule). However it’s not yet time to celebrate — there are still many issues that need to be addressed before the Make-based build system may retire.
Want to try? Checkout the GHC repository and run hadrian/build.sh -j or hadrian/build.bat -j on Windows and it should build you a GHC binary. In case of problems, have a look at the README and/or raise an issue.
Here is a quick update on the on-going development:
- Hadrian can build GHC and can already be used as part of the CI infrastructure. However, the resulting binary does not pass the validation. Zhen Zhang is looking into this, but more help is needed.
- A major refactoring by Moritz Angermann is on the way. Moritz is primarily interested in cross compilation, but to make it work he had to get rid of the
ghc-cabalutility, reorganise the build tree, and make numerous other improvements to Hadrian. - There is currently no support for binary distribution. Ben Gamari is looking into this issue.
- Dynamic linking on Windows is not implemented. Tamar Christina has kindly offered help with this.
- Hadrian source code is still not fully documented and tested, and generally requires some polishing. I am currently taking care of this when not distracted by urgent bug fixes and will appreciate your help in making Hadrian easier to understand and use.
I can’t believe that we seem to approach the finish line! It’s been a long, tedious but also interesting project. Thank you all for helping us get this far, and I hope we’ll celebrate the switch from Make to Hadrian soon.
Towards Cloud Build Systems with Dynamic Dependency Graphs
I’ve recently submitted an application to the Royal Society Industry Fellowship scheme, aiming to continue my journey into the world of build systems. Below is the technical section, which I think is worth sharing regardless of the outcome of my application. I’d like to thank Neil Mitchell, Simon Peyton Jones, Simon Marlow and Nick Benton for many enlightening discussions on build systems that helped me understand both the problem and possible approaches to solving it.
Continue reading Towards Cloud Build Systems with Dynamic Dependency Graphs
Build GHC on Windows using Hadrian and Stack
To build GHC on Windows you usually need to jump through a lot of hoops, which may be confusing even for experienced GHC developers.
Hadrian to the rescue!
Hadrian, a new build system for GHC that I’ve been developing, is written in Haskell and can therefore be built and run via Stack that can install appropriate bootstrapping GHC and MSYS2 environment in an automated and robust way. This was first pointed out by Neil Mitchell, and I’ve recently simplified build instructions even further. The latest version of the instructions is maintained here.
Continue reading Build GHC on Windows using Hadrian and Stack
Shaking up GHC
As part of my 6-month research secondment to Microsoft Research in Cambridge I am taking up the challenge of migrating the current GHC build system based on standard make into a new and (hopefully) better one based on Shake. If you are curious about the project you can find more details on the wiki page.
During this week I’ve been trying to wrap my head around the current build system and so far I’ve felt rather intimidated by the scale of this undertaking. Yet I am optimistic because I’m helped by several much smarter people 🙂 If you are a Haskell fan and would like to help make GHC better, please join this team effort!