{"id":247,"date":"2016-10-24T01:36:04","date_gmt":"2016-10-24T00:36:04","guid":{"rendered":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/?p=247"},"modified":"2018-07-07T11:58:14","modified_gmt":"2018-07-07T10:58:14","slug":"cloud-and-dynamic-builds","status":"publish","type":"post","link":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/cloud-and-dynamic-builds\/","title":{"rendered":"Towards Cloud Build Systems with Dynamic Dependency Graphs"},"content":{"rendered":"<p>I&#8217;ve recently submitted an application to <a href=\"https:\/\/royalsociety.org\/grants-schemes-awards\/grants\/industry-fellowship\/\">the Royal Society Industry Fellowship scheme<\/a>, aiming to continue <a href=\"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/shaking-up-ghc\/\">my journey into the world of build systems<\/a>. Below is the technical section, which I think is worth sharing regardless of the outcome of my application. I&#8217;d like to thank Neil Mitchell, Simon Peyton Jones, Simon Marlow and Nick Benton for many enlightening discussions on build systems that helped me understand both the problem and possible approaches to solving it.<\/p>\n<p><!--more--><\/p>\n<p><strong>Build systems<\/strong><\/p>\n<p>A <em>build system<\/em> is a critical component of most software projects, responsible for compiling the source code written in various programming languages and producing executable programs &#8212; end software products. Build systems sound simple, but they are not; in large software projects the build system grows from simple beginnings into a huge, complex engineering artefact. Why? Because it evolves every day as new features are continuously being added by software developers; because it builds programs for a huge variety of target hardware configurations (from mobile to cloud); and because it operates under strict correctness and performance requirements, standing on the critical path between the development of a new feature and its deployment into production.<\/p>\n<p>It is known that build systems can take up to 27% of software development effort, and that improvements to build systems rapidly pay off [1]. Despite its importance, this subject is severely under-researched, which prompts major companies, such as Facebook and Google, to invest significant internal resources to make their own bespoke build system frameworks.<\/p>\n<p><strong>Challenges in large-scale build systems<\/strong><\/p>\n<p>The following two requirements are important for complex large-scale build systems, however, to the best of my knowledge, no existing build system can support both:<\/p>\n<ul>\n<li><strong>Distributed cloud builds:<\/strong> codebase of large software projects comprises millions of lines of code spread across thousands of files, each built independently by thousands of developers on thousands of machines. A distributed cloud build system speeds up builds dramatically (and saves energy) by transparently sharing build products among developers.<\/li>\n<li><strong>Dynamic dependency graphs:<\/strong> many existing build systems need to know all the dependencies between software components, i.e. the <em>dependency graph<\/em>, at the start of the build process. This makes it possible to analyse the graph ahead of time, but is fundamentally limited: parts of the dependency graph can often be discovered only during the build process, i.e. the dependency graph is dynamic, not static.<\/li>\n<\/ul>\n<p>There exist build systems that support cloud builds, e.g. Facebook\u2019s <a href=\"https:\/\/buckbuild.com\/\">Buck<\/a>\u00a0and Google\u2019s <a href=\"https:\/\/www.bazel.io\/\">Bazel<\/a>, as well as build systems that support dynamic dependencies, e.g. Neil Mitchell&#8217;s\u00a0<a href=\"http:\/\/shakebuild.com\/\">Shake<\/a>\u00a0(originally developed at Standard Chartered) and Jane Street\u2019s <a href=\"https:\/\/github.com\/janestreet\/jenga\">Jenga<\/a>, but not both.<\/p>\n<p><strong>Proposed approach and research objectives<\/strong><\/p>\n<p>I will combine two known techniques when developing the new build system: 1) storing build results in a <em>cloud build cache<\/em>, where a file is identified by a unique hash that combines hashes of the file, its build dependencies and the environment, so that users of the build system can fetch an already built file from the cache, or build it themselves and share it with others by uploading it into the build cache; and 2) storing the last known version of the dependency graph in a <em>persistent graph storage<\/em>, and updating it whenever a file needs to be rebuilt and the newly discovered set of dependencies differs from the previous record, as implemented in the Shake build system [2].<\/p>\n<p>My first research objective is to <u>formalise the semantics of cloud build systems with dynamic dependency graphs<\/u>\u00a0in the presence of <em>build non-determinism<\/em> (real compilers are not always deterministic) and <em>partial builds<\/em> (not all intermediate build results are necessary to store locally). I will then <u>integrate cloud build caches with persistent graph storage<\/u>\u00a0and <u>develop a domain-specific language for the new build system<\/u>\u00a0using scalable functional programming abstractions, such as <em>polymorphic and monadic dependencies<\/em>, <em>high-level<\/em> <em>concurrency primitives<\/em>, and <em>compositional build configurations<\/em> [3]. In 2012-2014 I developed <em>an algebra for dependency graphs<\/em> [4] that I will use for proving the correctness of new build algorithms. The final research objective is to <u>evaluate the developed build system and explore opportunities for its integration into a real large-scale build infrastructure<\/u>.<\/p>\n<p><strong>References<\/strong><\/p>\n<p>[1] S. McIntosh et al. <em>&#8220;An empirical study of build maintenance effort&#8221;<\/em>, In Proceedings of the International Conference on Software Engineering (ICSE), ACM, 2011.<\/p>\n<p>[2] N. Mitchell. \u201c<a href=\"http:\/\/ndmitchell.com\/downloads\/paper-shake_before_building-10_sep_2012.pdf\"><em>Shake before building \u2013 replacing Make with Haskell<\/em><\/a>\u201d, In Proceedings of the International Conference on Functional Programming (ICFP), ACM, 2012.<\/p>\n<p>[3] A. Mokhov, N. Mitchell, S. Peyton Jones, S. Marlow. <em>\u201c<a href=\"https:\/\/www.staff.ncl.ac.uk\/andrey.mokhov\/Hadrian.pdf\">Non-recursive Make Considered Harmful: Build Systems at Scale<\/a>\u201d<\/em>, In Proceedings of the International Haskell Symposium, ACM, 2016.<\/p>\n<p>[4] A. Mokhov, V. Khomenko. <em>\u201c<a href=\"https:\/\/www.staff.ncl.ac.uk\/andrey.mokhov\/algebra.pdf\">Algebra of Parameterised Graphs<\/a>\u201d<\/em>, ACM Transactions on Embedded Computing Systems, 13(4s), p.143, 2014.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve recently submitted an application to the Royal Society Industry Fellowship scheme, aiming to continue my journey into the world of build systems. Below is the technical section, which I think is worth sharing regardless of the outcome of my application. I&#8217;d like to thank Neil Mitchell, Simon Peyton Jones, Simon Marlow and Nick Benton &hellip; <a href=\"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/cloud-and-dynamic-builds\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Towards Cloud Build Systems with Dynamic Dependency Graphs<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1174,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,12],"tags":[14],"class_list":["post-247","post","type-post","status-publish","format-standard","hentry","category-coding","category-research","tag-build"],"_links":{"self":[{"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/posts\/247","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/users\/1174"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/comments?post=247"}],"version-history":[{"count":16,"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/posts\/247\/revisions"}],"predecessor-version":[{"id":263,"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/posts\/247\/revisions\/263"}],"wp:attachment":[{"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/media?parent=247"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/categories?post=247"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/wp-json\/wp\/v2\/tags?post=247"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}