Managing Frontend Dependencies & Deployment Part 2: Gulp

In part 1 we looked at how to use Bower to install and maintain frontend libraries and their dependencies, this post looks at moving on from managing our dependencies to deployment using Gulp.

Gulp

Bower manages the versions of packages we have installed, and the dependencies of each package we wish to use. But how do we go about using those packages in our code efficiently — both for development and deployment?

Enter Gulp. Gulp is a task runner, which makes it a distant relative of the archaic Make, or tools like Ant, Phing, Rake, or the other new kid on the block, Grunt.

Gulp vs Grunt

There are two task runners that have gained popularity in this space over the last year, Gulp, and Grunt.

Grunt was the first to gain popularity, and tries to provide built-in functionality to cover the common use cases. It follows a configuration-based approach.

Gulp on the other hand provides very little out of the box, instead preferring to defer functionality to many small single-feature plugins. Gulp uses a streaming pipeline of plugins to create a complex workflow.

While both tools can run tasks in parallel, Gulp does so by default, attempting to achieve multiple concurrency — running as many tasks as possible at the same time while respecting things such as dependent tasks.

Four Things

Gulp does four things out of the box:

1. Define tasks with gulp.task()
2. Watch the file system for changes with gulp.watch()
3. Open files/directories with gulp.src()
4. Output files/directories with gulp.dest()

Gulp will call the default task, or any other task specified on the command line, automatically.

Everything else is achieved by calling consecutive pipe() calls on the result of gulp.src().

Virtual File System & Streaming

Gulp works on a virtual file system, known as vinylfs, which sits on top of the vinyl virtual file format. This means you can modify files without touching the disk until you are finished — allowing gulp to do its multi-pipe streaming without having to write to temporary files.

To learn more about streaming, read the Stream Handbook.

Installation

Gulp installation is identical to Bower, to install globally:

$ npm install -g gulp

To install locally, and save to our package.json:

$ npm install --save-dev gulp

Creating our Workflow

Let’s say we want create a single site-wide CSS and site-wide JS file that will automatically be included in our template. We also need the ability to easily switch to the original files for debugging.

Our workflow has two goals. Let’s look at the first, minify/concatenation:

1. Find all the files being used
2. Minify them
3. Concat them
4. Save them
5. Replace the references in our templates

To do this, we will use the gulp-uglifyjs, gulp-minify-css and gulp-usemin packages. To install them simply do:

$ npm install --save-dev gulp-usemin gulp-uglify gulp-minify-css

Our workflow then might look something like this:

1. gulp.src()
2. uglifyjs and minify-css with:
3. concat options
4. gulp.dest()
5. usemin (replace)

For this, let’s assume that our template currently resides in /src/templates/layout.tpl. First, we will copy this to: /src/templates/layout.src.tpl. This file contains the information for gulp to work on, generating our layout.tpl for production or development, as appropriate.

Next, let’s add some directives to our template for usemin to work with. We do this by placing special comments around our CSS and Javascript blocks, like so:

<!-- build:css /css/site.css -->
<link href="/bower_components/bootstrap/dist/css/bootstrap.css" rel="stylesheet">
<link href="/bower_components/bootstrap/dist/css/bootstrap-theme.css" rel="stylesheet">
<!-- endbuild -->

and for our Javascript:

<!-- build:js /js/site.js -->
<script type="text/javascript" src="/bower_components/jquery/dist/jquery.js"></script>
<script type="text/javascript" src="/bower_components/bootstrap/dist/js/bootstrap.js"></script>
<!-- endbuild -->

Next, let’s build out our tasks. We do this in gulpfile.js. First, we need to pull in all of our required modules:

var gulp = require('gulp');
var usemin = require('gulp-usemin');
var uglify = require('gulp-uglify');
var minifyCss = require('gulp-minify-css');

Next, we define our default task:

gulp.task('default', function() {
   gulp.src('src/templates/layout.src.tpl')
        .pipe(usemin({
            assetsDir: 'public',
            css: [minifyCss(), 'concat'],
            js: [uglify(), 'concat']
        }))
        .pipe(gulp.dest('public'));
});

Stepping through this line by line, we define our task with gulp.task(), called default, and with a callback function.

We then open our src/templates/layout.src.tpl with gulp.src().

Next, we pipe() it to usemin, with a configuration that specifies the location of the assets used in our templates (assetsDir) and then how we want to handle our CSS and Javascript files — with minifyCss() or uglify() and passing in the 'concat' argument.

Finally we pipe() to gulp.dest() to save all the files.

To run this, simply call gulp on the command line:

$ gulp
[09:11:05] Using gulpfile /path/to/gulpfile.js
[09:11:05] Starting 'default'...
[09:11:07] Finished 'default' after 2.01 s

However, we have two issues. The template is copied to public/layout.src.tpl, not app/templates/layout.tpl, and we are missing the bootstrap font resources.

To fix this, let’s build some tasks, first a fix-template task, which will use the gulp-rename and gulp-rimraf plugin. gulp-rename will rename the file opened by gulp.src in the virtual vinylfs while gulp-rimraf will remove the original file from disk. We then output the in-memory file to its new location.

var rename = require('gulp-rename');
var rimraf = require('gulp-rimraf');

gulp.task('fix-template', ['minify'], function() {
    return gulp.src('public/layout.src.tpl')
        .pipe(rimraf())
        .pipe(rename("layout.tpl"))
        .pipe(gulp.dest('src/templates'));
});

To make this automatically run, we could specify this task as a dependency for the default task, but that means it would run first, before the file is put in the wrong place, so instead, we have to do it the other way around.

First, let’s rename our default task to minify, by changing:

gulp.task('default', function() {

to:

gulp.task('minify', function() {

Then, add the minify task as dependency of the fix-template task, by specifying it as the second argument for gulp.task():

gulp.task('fix-template', ['minify'], function() {

We then run our tasks in the correct order by running gulp with:

$ gulp fix-template
[16:48:29] Using gulp file /path/to/gulpfile.js
[16:48:29] Starting 'minify'...
[16:48:29] Finished 'minify' after 44 ms
[16:48:29] Starting 'fix-template'...
[16:48:29] Finished 'fix-template' after 6.14 ms

This still doesn’t work as we’d expect however! Because our minify task is set to run asynchronously (the default is maximum concurrency), the dependency just requires that it be called, not that it be completed.

We can fix this in three ways, returning a valid stream, using a callback, or using a promise.

The simplest way is to use a return on the stream: simply prefix the first line of our task with return:

gulp.task('minify', function() {
   return gulp.src('src/templates/layout.src.tpl')
      ...

Seeing as the minify/fix-template is our default case, we can create a new empty default task with fix-template as a dependency and it will automatically run:

gulp.task('default', ['fix-template']);

Better yet, we can specify all our tasks here, so that gulp will attempt to run as many as possible:

gulp.task('default', ['minify', 'fix-template']);

This also means that should we ever resolve the dependency on minify for fix-template, then minify will still be called, also any other tasks that depend on minify can run immediately instead of depending on fix-template to be called.

The last thing we need to do is fix the bootstrap fonts. Currently they still reside in the public/bower_components/bootstrap/dist/fonts directory, but our site.css still points to the relative path ../fonts.

We can handle this two ways: we can copy the fonts to our public directory… or we can just update the file to point to the copy inside of our bower_components. To do this we use the simple gulp-replace plugin.

var replace = require('gulp-replace');

gulp.task('fix-paths', ['minify'], function() {
    gulp.src('public/css/site.css')
        .pipe(replace('../', '../bower_components/bootstrap/dist/'))
        .pipe(gulp.dest('public/css'));
});

Notice again how we have a dependency on the minify task; we should also add the task to our default dependencies:

gulp.task('default', ['minify', 'fix-template', 'fix-paths']);

Now, thanks to concurrency, the fix-template and fix-paths tasks will (potentially) both run concurrently after minify is completed.

One final thing we should add is a header to indicate the file has been auto-generated, and not to modify it directly. This can be achieved, by — you guessed it — gulp-header.

var header = require('gulp-header');

gulp.task('add-headers', ['fix-template'], function() {
    gulp.src('src/templates/layout.tpl')
        .pipe(header("<!-- This file is generated — do not edit by hand! -->\n"))
        .pipe(gulp.dest('src/templates'));

    gulp.src('public/js/site.js')
        .pipe(header("/* This file is generated — do not edit by hand! */\n"))
        .pipe(gulp.dest('public/js'));

    gulp.src('public/css/site.css')
        .pipe(header("/* This file is generated — do not edit by hand! */\n"))
        .pipe(gulp.dest('public/css'));
});

This time we need to depend on fix-template as the template file must be in its final location.

Development and Cleanup

Let’s create two more simple tasks, a clean task, and a dev task.

gulp.task('clean', function() {
    var generated = ['public/js/site.js', 'public/css/site.css', 'src/templates/layout.tpl'];
    return gulp.src(generated)
        .pipe(rimraf());
});

gulp.task('dev', ['clean'], function() {
    gulp.src('src/templates/layout.src.tpl')
        .pipe(rename('layout.tpl'))
        .pipe(gulp.dest('src/templates'));
});

The clean task simply deletes all the generated files, while the dev task copies layout.src.tpl to layout.tpl adding our auto-generated header — leaving the original bower_component paths in place.

We can also add the clean task as a dependency for the minify task:

gulp.task('minify', ['clean'], function() {

Now when we run gulp, we will see:

$ gulp
[22:56:15] Using gulpfile /path/to/gulpfile.js
[22:56:15] Starting 'clean'...
[22:56:15] Finished 'clean' after 46 ms
[22:56:15] Starting 'minify'...
[22:56:19] Finished 'minify' after 4.81 s
[22:56:19] Starting 'fix-template'...
[22:56:19] Starting 'fix-paths'...
[22:56:19] Finished 'fix-paths' after 8.52 ms
[22:56:19] Finished 'fix-template' after 36 ms
[22:56:19] Starting 'add-headers'...
[22:56:19] Finished 'add-headers' after 4.18 ms
[22:56:19] Starting 'default'...
[22:56:19] Finished 'default' after 24 μs

This is now our production deployment option.

Automation

If we want to automatically keep our minified files up-to-date during development, we can use the gulp.watch() functionality. Let’s create our final task, watch:

gulp.task('watch', ['default'], function() {
    var watchFiles = [
        'src/templates/layout.src.tpl',
        'public/bower_components/*/dist/js/*.js',
        '!public/bower_components/*/dist/js/*.min.js',
        'public/bower_components/*/dist/*.js',
        'public/bower_components/*/dist/css/*.css',
        '!public/bower_components/*/dist/css/*.min.css',
        'public/bower_components/*/dist/font/*'
    ];

    gulp.watch(watchFiles, ['default']);
});

Here we are watching our templates, as well as all Bower package .js and .css files. Note that we exclude .min.js and .min.css files.

We then call gulp.watch() passing in our array of files, and the task we want to run when changes are detected: default.

$ gulp watch
[23:05:01] Using gulpfile /path/to/gulpfile.js
[23:05:01] Starting 'watch'...
[23:05:01] Finished 'watch' after 30 ms

At which point gulp will sit and wait for changes.

If you want to make sure that the task runs on start up, you can set the default task as a dependency:

gulp.task('watch', ['default'], function() {

Automating the Automation

One thing you may have noticed is the large number of require calls we need to make to include all of our needed plugins.

We can shorten this to just one, using — ironically — another plugin, gulp-load-plugins.

This plugin will automatically load all gulp plugins from our package.json using lazy-loading, making them all accessible via a single object.

$ = require('gulp-load-plugins')(); // Note the extra parens

From this point, all our plugins will be available as $.<plugin>, stripping the gulp- and using camel-case naming. This means that gulp-usemin and gulp-uglify become $.usemin and $.uglify respectively, and gulp-minify-css becomes $.uglifyCss.

Review

You can see the completed gulpfile.js and other related files, in this gist.

Note: This is example code, not intended for production!

Take a Breath

The frontend tool chain is still very much under development. It is definitely standing on the shoulders of giants, like Composer, Bundler, and particularly npm. Hopefully, it will one day become a giant in its own right.

Being node.js tools, they are typically asynchronous, enabling them to perform with maximum concurrency, as fast as possible — which is critical for speedy deployment.

Get Up and Running

Now that we are using gulp and Bower, in the next part of this series, we will look at Yeoman — a tool for automating the scaffolding when you create your next application.