Building web applications with Make

A Makefile

Make is a build automation tool originally designed to compile source code into executable programs. However, because it is such a proven, flexible and versatile tool, it is also a great fit for other applications. I have been using Make to automate build tasks for web applications for a few years now with great satisfaction, and in this article I will show some of its interesting features and some makefile snippets I commonly use in the projects I work on.

Introduction

Historically, a "build" in the context of software development referred to the process of compiling source code into machine code that could be executed. However, with the popularity of interpreted languages (such as JavaScript, Python and PHP) that don't require compilation in order to run, together with modern practices such as continuous integration, the definition of a software build has been expanded to include running automated tests, static analysis tools and other quality assurance steps. Also, in the context of web applications, even an application written mainly in an interpreted programming language may still contain frontend source code (such as JavaScript and CSS) which needs to be minified, concatenated or otherwise preprocessed before it is deployed to production.

As the complexity of a software project grows, it becomes more and more cumbersome to perform all these steps by hand. Not only do they need to be documented so everyone does it the same way, but they may also change over time as the project evolves. Forgetting a step or executing a wrong command can cause unexpected results. To speed up the build process, reduce the risk of human errors and allow for changes over time, the build process should be automated and the automation scripts should be committed as part of the project's source code.

Although in its simplest form this could be a shell script containing a list of commands required to build the application, it is common practice to use a dedicated build automation tool. Some of the more popular options include Apache Ant and Apache Maven (both popular for Java development), Gradle (Java), Grunt (JavaScript), Gulp (JavaScript), Phing (PHP), Rake (Ruby), and Make.

Introduction to Make

Make has been created in 1976 and has since become a widespread tool. It is actually a family of tools, as there are a couple of different implementations of Make, such as BSD Make, GNU Make and Microsoft nmake, each with their own enhancements and features. In this article I will focus on GNU Make, the standard implementation of Make on Linux and OS X.

Make uses makefiles to understand the relationships between source files, output files, and the commands that are required to create these output files. Make will look for a file called Makefile in the current directory (unless another filename is specified).

Based on the information provided in the makefile, Make will automatically determine whether an output file (called target) is out of date and needs to be rebuilt by comparing its file modification time with that of its source files (prerequisites). If the target doesn't exist yet, it will always be built.

When you run Make, you can provide an optional list of target names that should be built:

make [TARGET ...]

If you don't specify a target, Make will build the default target (more about this later).

Rule syntax

The syntax of a basic rule looks like this:

target [target ...]: [prerequisite ...]
    recipe

A target is usually the name of an output file that can be created by executing the recipe. A rule can have multiple targets, which can be useful when multiple files can be created with the same (or similar) command.

A prerequisite is a file that is used as input to create a target. As I noted earlier, Make compares the timestamps of the target and its prerequisites to determine whether the target is up to date or should be rebuilt. A target can have zero, one or many prerequisites.

A recipe is the command that Make should execute to build the target. A recipe can also consist of multiple commands, which may be spread over multiple lines. Note that each recipe line should be indented with a tab character.

Installing dependencies using a package manager

A common build task for modern web applications is to install third-party dependencies using a package manager, such as Composer or NPM. My first (naive) approach to install dependencies using Make was to have an install target in my Makefile to execute the install commands of the different package managers. I would run make install whenever I checked out a project for the first time or after switching branches. The install target would look like this:

install:
    composer install
    npm install

Because a file or directory called install would never be created, this was effectively a phony target (although I didn't know that term by then) and Make would rerun these commands every time I ran make install, even if there was nothing to install or update. I also had to always remember running this before any other Make targets, because none of my other targets had install as their prerequisite.

My first improvement was to split up the install target and create targets named after the directories containing dependencies (vendor, node_modules, etc.), with the dependency and lock files as prerequisites:

vendor: composer.json composer.lock
    composer install

node_modules: package.json package-lock.json
    npm install

This worked nicely because Make would now compare the timestamp of the package directory with that of the dependency files, and only execute the recipe when the directory didn't exist or was out of date with one of the dependency files. I also started to add these targets as prerequisites to other rules that depended on (up-to-date) packages in order to run.

Later, I realised that I could drop the file listing the requirements (composer.json / package.json) and only use the lock file as the prerequisite. The idea is that the lock file will change whenever a dependency is installed or updated. The file listing the requirements often also contains other metadata, which can be changed without requiring an update of the installed packages. So, my rules now look like this:

vendor: composer.lock
    composer install

node_modules: package-lock.json
    npm install

I'm currently experimenting to further improve these rules by replacing them with pattern rules that match with any file within the directory, so I can use them as prerequisites for targets that depend on a specific file from a package. I'm not really sure about this move yet, but I will update this article if it happens to work out (or not).

As a final remark: if a prerequisite is missing and there is no rule to create it, Make will fail. So for projects (mainly libraries) where I add the lock files to the .gitignore, I often use Make's wildcard function to treat the lock file as an "optional prerequisite":

vendor: $(wildcard composer.lock)
    composer install

Note that this optional prerequisite is not really a Make concept. It's just a way to have a prerequisite that will not let Make fail when it doesn't exist.

Using a default target

The documentation of GNU Make lists a number of conventions for writing Makefiles for GNU programs. Not all of them can be applied to the context of web application development, but I've adopted some that make sense for web applications. One of these is to have every makefile contain an all target, which the documentation describes as:

Compile the entire program. This should be the default target. (...)

Being the default target refers to the fact that Make, when called without arguments, will build the first target that appears in the makefile. By adopting this convention I can enter any project, run make (or make all), and I have a working application! Depending on the specific requirements of the project I add prerequisites to the all target so that dependencies are installed, assets are compiled, etc:

.PHONY: all
all: vendor public/app.js public/app.css

The .PHONY: all tells Make that all should be treated as a phony target, meaning a target that is not really the name of a file. This ensures that Make will execute the recipe, even if a file or directory with the same name happens to exist already.

Compiling Sass or LESS files

When I work on a fairly simple project that requires some Sass but not a full-blown Webpack setup, I prefer to run the sass executable directly. Compiling Sass files (or LESS for that matter) is a good use case for Make because of the clear relationship between source (Sass) and output (CSS) files, which allows you to take the most advantage of Make's out of date calculation:

app.css: app.scss
    sass app.scss app.css

Now when you run make app.css, Make will only compile the Sass file when it is newer than the compiled CSS.

To avoid repetition and make this rule more flexible, we can use automatic variables that contain the names of the target and the prerequisite:

app.css: app.scss
    sass $< $@

$< contains the name of the first prerequisite (in this case app.scss), and $@ contains the name of the target, which in this case is app.css.

In real-life applications my Sass files often import other Sass files, which can either be my own (partials) or maintained by a third party (for example, the Bootstrap Sass files). To ensure that when these are updated my output CSS is rebuilt, I list them as prerequisites as well:

app.css: sass/app.scss sass/_*.scss node_modules
    sass $< $@

Note that this is why I'm using the $< variable (which contains only the first prerequisite): I want Make to rebuild app.css if any of the prerequisites are newer than the target, but I only want the first prerequisite (sass/app.scss) to be passed as an argument to the sass executable.

The last examples assumed that there is only a single CSS file to be built. However, in some cases I want to create a generic rule that can be used to build multiple targets. In these cases pattern rules are quite useful:

%.css: sass/%.scss sass/_*.scss node_modules
    sass $< $@

A pattern rule is a rule where the target contains the % character. The % character can match any nonempty string. If a prerequisite also contains the % character, it will be replaced with the part of the target that matched the % character (called the stem). For example, when calling make print.css with the above rule, its first prerequisite will be sass/print.scss.

Last but not least, sass accepts a number of options which can be used to change the generated CSS. For instance, when building for production I might want to minify the generated CSS, while during development I would rather want expanded CSS and source mapping. By creating a variable SASSFLAGS and using that in the recipe, I have a recipe that can be reused but produces different output for development and production:

SASSFLAGS = --style expanded

%.css: sass/%.scss sass/_*.scss node_modules
    sass $(SASSFLAGS) $< $@

This variable can be overridden when building for production, in order to get minified output. This can be done by either setting the variable on the command line when calling Make (i.e. make SASSFLAGS='--style compressed --sourcemap=none' app.css) or by setting a target-specific variable (more about that later). The only issue I haven't really been able to solve yet (without building assets for production in a separate directory or giving them different filenames) is that when the output file already exists, Make will not re-run the recipe.

Creating favicons from source images

Again, when working on a small project that doesn't require a complex JavaScript asset build system, I sometimes just want to generate a set of favicons from a source image. To do so, I use convert (part of ImageMagick) to resize the input image into 16x16, 32x32 and 48x48 pixel PNG images which are padded with a background color to make them square, which are then combined into a favicon.ico file using icotool (part of the icoutils package):

favicons = public/favicon.ico public/favicon-16x16.png public/favicon-32x32.png public/apple-touch-icon.png

all: public/app.js public/app.css $(favicons)

public:
    mkdir -p $@

public/favicon-%.png: images/portrait.png | public
    convert $< -resize $* -background white -gravity center -extent $* $@

public/apple-touch-icon.png: images/portrait.png | public
    convert $< -resize 180x180 -background white -gravity center -extent 180x180 $@

public/favicon.ico: public/favicon-16x16.png public/favicon-32x32.png public/favicon-48x48.png | public
    icotool --create --output=$@ $^

At the top of the makefile I create a variable favicons which holds the names of the favicon files I need to generate, so I can use that as the prerequisites for another target (such as all).

I then have the various rules for creating the PNG and ICO files. In these rules I use two automatic variables I haven't discussed yet: $*, which contains the stem from my pattern rule (the part of the target that matches with the % character), and $^ which contains the names of all prerequisites (with spaces between them).

The public prerequisite is an order-only prerequisite (denoted by the pipe symbol | between the normal prerequisites and order-only prerequisites). This means that the build directory will be created if it doesn't already exist, but the files inside it don't have to be recreated whenever the timestamp of the directory changes.

Of course this example can be improved to generate even more favicon files to support higher resolution icons that are used by different browsers and operating systems. The FAQ from RealFaviconGenerator.net contains a good list of recommended favicon files.

Compiling frontend assets with Webpack

When I do work on a project that uses Webpack or a similar tool to build frontend assets, I still like to add these tools to my Makefile instead of running them directly. This helps integrating them in the rest of the application's build process (so I can build the whole application with only one command) while at the same time making sure that they will only run when one of the source files has actually changed.

assets = public/main.css public/main.js public/favicon.ico

$(assets): sass/* src/*.js images/* node_modules
    node_modules/.bin/webpack

The assets variable conveniently contains all assets that need to be built for this project, and can be used as prerequisite for other targets such as the default all target.

Building a deployable tar archive

In order to create a build artifact that can be deployed to a remote server, I again use one of the standard targets from GNU's Makefile conventions: dist. Running make dist should create a tarball containing all the files required to run the application in production. I then use Ansible to copy this archive to the server, extract it into a new release directory, switch symlinks, etc.

sources = config/ public/ src/ templates/
version = $(shell git describe --tags --dirty --always)
build_name = my-app-$(version)
build_dir = build/my-app-$(version)

# (other targets)

.PHONY: dist
dist: SASSFLAGS = --style compressed --sourcemap=none
dist: all
    -rm -rf $(build_dir)
    mkdir -p $(build_dir)
    ln --symbolic --relative $(sources) composer.json composer.lock $(build_dir)
    composer install --working-dir=$(build_dir) --no-dev
    rm $(build_dir)/composer.json $(build_dir)/composer.lock
    tar --create --dereference --gzip --file=build/$(build_name).tar.gz --directory=build $(build_name)
    -rm -rf $(build_dir)

At the top of the Makefile, I set a couple of variables which can be reused throughout the file. sources contains a list of source files and/or directories. version runs a Git command to describe the current state of the workspace. Especially when using tags this is very useful, as it will return the most recent tag and some other information to identify this build. build_name contains a name for this build based on the name of the project and the value of version, and build_dir contains the name of a temporary directory which will be archived to produce the build artifact. This allows to filter development dependencies out of the build artifact without affecting the current working directory.

Remember how I created a SASSFLAGS variable in order to pass different options to Sass depending on whether I was building for development or production? Using a target-specific variable value, I can override the value specifically for the dist target, and this value is even passed to rules that are prerequisites of dist. This way when I run any other target the global value of SASSFLAGS applies, but when I run make dist the value is overwritten with one that results in compressed CSS without source maps.

The recipe itself contains the following steps:

  1. The temporary directory for this build is removed (if it exists already) and re-created. Note that the dash in front of the rm command tells Make to ignore any errors, so Make will not fail if the directory does not exist.
  2. Symbolic links are created to the files and directories listed in the sources variable.
  3. Composer is used to install the production dependencies inside the build directory. Afterwards, the composer.json and composer.lock files are removed.
  4. A gzip-compressed tarball is created containing the build directory and its contents (the --dereference flag is used to follow the symlinks we made earlier, instead of having actual symlinks in the archive).
  5. The temporary directory is cleaned up.

To prevent the the contents of the build directory and build artifacts from accidentally being committed to Git, the build directory is included in the .gitignore file of the project.

Running automated tests

Although this normally doesn't result in files being created, I find it very convenient to also include test targets in my Makefiles. For most projects I create a test target to run the full test suite, which has more fine-grained targets as its prerequisites. This way I can easily run the whole suite with make test or run a smaller part of the test suite, for instance make unit-tests:

.PHONY: test lint unit-tests acceptance-tests coding-standards security-tests
test: lint unit-tests acceptance-tests coding-standards security-tests

lint: vendor
    vendor/bin/parallel-lint $(sources)
    bin/console lint:twig templates/
    vendor/bin/phpstan analyse --level=7 $(sources)

unit-tests: vendor
    vendor/bin/phpunit

acceptance-tests: vendor
    vendor/bin/behat -v

coding-standards: vendor
    vendor/bin/phpcs -p --colors

security-tests: vendor
    vendor/bin/security-checker security:check
    npm audit

Of course all these targets are phony targets. Note that for every target I list the required prerequisites, so I can always run the tests with a single command even if I just did a fresh checkout of the code.

Cleaning up

A last convention I've adopted from GNU's Makefile conventions is to always include a clean target in my Makefiles. Although the targets in the Makefile should always contain the right prerequisites to ensure that it will be rebuilt when it gets out of date, it can sometimes just be very convenient to remove all the files that were built by the Makefile and start with a clean slate.

.PHONY: clean
clean:
    rm -rf build node_modules vendor $(assets)

The Makefile conventions also list other cleanup targets (uninstall, distclean, mostlyclean and maintainer-clean), but I haven't really found a use case for these yet in my projects so I haven't implemented them yet.

Using Prism for syntax highlighting with Sculpin

Update: since publishing this article I have replaced Prism with Pygments, a Python package that is run when building the Sculpin site so that JavaScript is no longer required to view my site. Although this removes the original issue that inspired me to write this article, the solution can still be useful for other people, so I've decided to keep this article on my blog.

I have been playing with the idea of starting my own blog for a while, but until now my website consisted of just a static index.html (along with some Sass file and images). Writing the HTML manually worked fine for a single landing page, but I realised that for a blog this would quickly become cumbersome. In order to keep the management of my site as easy as possible, without having to run a fully-fledged, database-backed content management system, I decided to go for a static site generator.

When it comes to static site generators, there are a lot of options available, written in many different programming languages. I decided to go with Sculpin, a static site generator written in PHP and based on Symfony. Just as most static site generators, Sculpin allows converting lightweight markup languages such as Markdown and Textile to HTML. Together with a template (written in Twig) this is rendered into a full HTML page. If you want to read more about Sculpin, my former colleague Matthias Noback has written some excellent blog posts about it.

After having set up Sculpin, a basic layout based on Bootstrap and some pages, the result was starting to look like a proper blog. However, obviously a blog about programming is not complete without code samples and syntax highlighting. At first I installed highlight.js for syntax highlighting, as I already had some limited experience with it from creating a presentation with code samples using reveal.js. However, after playing around with it for a bit, I discovered that the functionalities of highlight are a bit limited as it lacks support for certain syntax elements such as variable and method names.

Disappointed, I started searching for an alternative, and found Prism. Like highlight.js, Prism is a JavaScript library that offers syntax highlighting for a lot of different languages and supports multiple themes. However, the syntax highlighting offered by Prism seems far more complete than that of highlight.js, and when using the Darcula theme it displays my piece of test PHP code almost exactly the same as PhpStorm does:

<?php
declare(strict_types=1);

namespace NicWortel\Demo;

class Foo
{
    private const MY_CONST = 1;

    /**
     * @var string
     */
    private $bar = '';

    public function __construct(string $bar)
    {
        $this->bar = $bar;
    }

    public function getBar(): string
    {
        return $this->bar;
    }

    public function getFoo(): string
    {
        echo self::MY_CONST;

        return $this->getBar();
    }
}

But in order to get Prism working with my Sculpin blog, there was one more hurdle to take. The Markdown parser used by Sculpin surrounds code blocks (either indented or "fenced") with <pre><code> tags. When using fenced code blocks, you can add the language name after the three backticks which will translate into a class being set on the <code> tag:

```javascript
var s = "JavaScript syntax highlighting";
```

<pre><code class="javascript">var s = "JavaScript syntax highlighting";</code></pre>

While this works fine for highlight.js (which will even try to detect the language automatically if the class is omitted), Prism is a lot more strict and only works when code blocks have a language-xxx (or lang-xxx) class. Unfortunately, this doesn't play nicely with Markdown. I could decide to use ```language-xxx in my Markdown files, but this would disable all syntax highlighting within my Markdown files in PhpStorm, GitHub, etc.

Luckily PHP Markdown, the Markdown parser used by Sculpin, supports configuring a class prefix for fenced code blocks by setting the code_class_prefix property on the MarkdownExtra class. So the only remaining question was how to change the value of this property on the instance of MarkdownExtra used by Sculpin. And because Sculpin is based on the Symfony framework, this was actually quite easy. Just like Symfony, Sculpin can be configured using YAML files. In order to change the value of the code_class_prefix property, all I had to do was to create a file app/config/sculpin_services.yml and override the service definition of the Markdown parser:

# app/config/sculpin_services.yml
services:
  sculpin_markdown.parser:
    class: '%sculpin_markdown.parser.class%'
    properties:
      code_class_prefix: language-

And that was it! The Markdown parser now prepends all code classes in the generated HTML with language-, enabling syntax highlighting with Prism while keeping syntax highlighting inside the source Markdown files intact.