Template Engine - Refresh Template on File Change, and Prevent Circular References

Our Template Engine gives content authors the flexibility of writing in a authoring format (Markdown) and creating layout in a template format (Pug). However, as it stands there are a couple of issues:

  1. Because we are caching the templates, we need to refresh the templates when the underlying content changes, but right now we are not doing that.
  2. Since we allow for templates to include each other, it's important to catch circular references, or the it would become an infinite include.

Let's address each issues in turn to improve our Template Engine.

Refresh Templates on File Change

For content authors, the ability to change contents and see the changes reflected is a big deal, since most content usually go through multiple modifications before they become the published draft. Without the ability to auto refresh it means that content authors would have to restart the service (like what we have been doing so far) with every content change, and that would render the system unusable (we've survived the numbers of npm start so far, but we'll address it in the near future as well).

There are two alternative approaches we can take:

  1. In getTemplate, keep track of the last modified timestamp of the file. If the next getTemplate returns a later timestamp, it means it's time to refresh the template.

  2. With each getTemplate, register the file path of the template with a file watcher to watch for changes. When the file watcher signals change, refresh the template.

The first approach is an easier approach, however, it has the cost of having to perform file system access every time the template is accessed. The second approach is conceptually more complex but the cost of IO is only paid when the underlying files are actually changed. We will show implementations for both.

Keep Track of the Last Modified Timestamp

The best place to keep track of the timestamp is with the template object that we are tracking. To do so we'll modify the Template type as follows:

interface Template {
  filename: string;
  fullPath: string;
  lastModified: number;
  fn: (locals ?: {[key: string] : any}) => string;
}

And we'll need to change compileTemplate to return the equivalent object structure instead of just a function:

  compileTemplate(fullTemplateFilePath : string) : Promise<Template> {
    let extname = path.extname(fullTemplateFilePath);
    switch (extname) {
      case '.pug':
        return this.compilePugTemplate(fullTemplateFilePath);
      case '.md':
        return this.compileMarkdownTemplate(fullTemplateFilePath);
      default:
        throw new Error(`UnknownTemplateType: ${extname}`);
    }
  }

With the refactored out of compiledPugTemplate and compileMarkdownTemplate as follows:

  compilePugTemplate(fullTemplateFilePath : string) : Promise<Template> {
    return fs.statAsync(fullTemplateFilePath)
      .then((stat) => {
        if (stat.isFile()) {
          return {
            filename: path.basename(fullTemplateFilePath),
            fullPath: fullTemplateFilePath,
            mtime: stat.mtime.getTime(),
            fn: pug.compileFile(fullTemplateFilePath, {
              filename: path.basename(fullTemplateFilePath),
              basedir: this._basePath
            })
          }
        } else {
          throw new Error(`NotFile: ${fullTemplateFilePath}`)
        }
      });
  }

  compileMarkdownTemplate(fullTemplateFilePath : string) : Promise<Template> {
    return fs.statAsync(fullTemplateFilePath)
      .then((stat) => {
        if (stat.isFile()) {
          return fs.readFileAsync(fullTemplateFilePath, 'utf8')
            .then((data) => {
              return {
                filename: path.basename(fullTemplateFilePath),
                fullPath: fullTemplateFilePath,
                mtime: stat.mtime.getTime(),
                fn: (locals: any) => {
                  return md.render(data, { html : true })
                }
              }
            })
        } else {
          throw new Error(`NotFile: ${fullTemplateFilePath}`)
        }
      })
  }

The calling of the template also needs to change, since it's no longer just a function:

  render(templateFilePath : string, locals ?: {[key: string]: any}) : Promise<string> {
    let fullTemplateFilePath = this.getFullTemplatePath(templateFilePath);
    return this.getTemplate(templateFilePath)
      .then((template) => {
        let result = template.fn(locals);
        // ... same as before ...
      })
  }

The above changes enable us to finally compare for the last modified timestamp in getTemplate:

  compileSetTemplate(fullTemplateFilePath : string) : Promise<Template> {
    return this.compileTemplate(fullTemplateFilePath)
      .then((template) => {
        this._templateMap[fullTemplateFilePath] = template;
        return template;
      })
  }

  getTemplate(templateFilePath : string) : Promise<Template> {
    let fullTemplateFilePath = this.getFullTemplatePath(templateFilePath);
    if (this._templateMap[fullTemplateFilePath]) {
      return fs.statAsync(fullTemplateFilePath)
        .then((stat) => {
          if (stat.mtime.getTime() > this._templateMap[fullTemplateFilePath].mtime) {
            return this.compileSetTemplate(fullTemplateFilePath);
          } else {
            return Promise.resolve<Template>(this._templateMap[fullTemplateFilePath]);
          }
        })
    } else {
      return this.compileSetTemplate(fullTemplateFilePath);
    }
  }

With this change in place our templates refreshes when they get modified.

Implement File Watcher To Watch for File Change.

As stated earlier, while the first approach is easy to understand and implement, we are paying the cost of revisiting the file system with every template access. It would have been nice to do it the other way around, i.e. let the file system notify us when the files have changed, and we refresh the templates only when we received the notification.

Node.js does have the ability to watch for file system, however, this is a known and curious sore spot of Node.js. Instead of making use of Node.js' built-in API, we'll make use of the npm package chokidar, which is designed to improve on Node.js' file monitoring capabilities. It is used by many popular tools that need to monitor files (gulp, karma, pm2, browserify, webpack), so is well battle tested.

First install the package:

npm install --save-dev @types/chokidar
npm install --save chokidar

Then import it:

import * as chokidar from 'chokidar';

The basic usage for chokidar is as follows:

let watcher = chokidar.watch(<path>, {
  ignored: /(^|[\/\\])\../,
  persistent: true
})
watcher.on('add', (path) => /* do something */)
  .on('change', (path, stat) => /* do something */)
  .on('unlink', (path) => /* do something */)

chokidar can watch individual files or a directory and its descendents. To simplify things, we'll watch a top level directory, which we have as _basePath in the TemplateEngine object.

class TemplateEngine {
  ...
  private _watcher : chokidar.FSWatcher;

  constructor(basePath : string) {
    ...
    this._watcher = chokidar.watch(this._basePath, {
      ignored: /(^|[\/\\])\../,
      persistent: true
    })
  }
  ...
}

Then we'll handle the change and unlink event (we don't need to handle add, since we only load template via getTemplate) as follows:

class TemplateEngine {
  ...
  private _watcher : chokidar.FSWatcher;

  constructor(basePath : string) {
    ...
    this._watcher
      .on('change', (fullPath : string, stat : fs.Stats) => {
        if (this._templateMap[fullPath]) { // only refresh if it exists in the template.
          this.compileSetTemplate(fullPath)
            .then(() => null)
        }
      })
      .on('unlink', (fullPath : string) => {
        delete this._templateMap[fullPath];
      })
  }
  ...
}

When we receive a change event, if it's a full path that we have currently compiled the code for - recompile the template. If it's an unlink event, delete the entry from _templateMap.

Keep in mind that since we are only watching _basePath, our current folder structures won't work correctly, as views and static are separate. So we'll need to either move them together, or watch at a higher level. Let's move them together.

# in project root dir.
git mv ./views/* ./static
let templateEngine = new TemplateEngine(path.join(__dirname, 'static')); // change from views to static.

And both jumbotron.md and call-to-action.md need to be updated with:

<Include>../signup.pug</Include>

Then our watcher code would watch over both the Pug and Markdown files together under one directory.

We can then remove the fs.statAsync check in getTemplate without loss of functionality.

  getTemplate(templateFilePath : string) : Promise<Template> {
    let fullTemplateFilePath = this.getFullTemplatePath(templateFilePath);
    if (this._templateMap[fullTemplateFilePath]) {
      return Promise.resolve<Template>(this._templateMap[fullTemplateFilePath]);
    } else {
      return this.compileSetTemplate(fullTemplateFilePath);
    }
  }

Prevent Circular References of Templates

Finally, we need to prevent circular references of templates, as our current code will run in an infinite loop (since we are running asynchronous code, it won't cause a stack overflow).

To detect a circular reference, we should have a stack of the previously seen template paths, so we can check to see if we are seeing a repeat. In the case that there is a repeat, we should return something that we can see visually telling us that we have created a cycle. This is so that content authors don't have to access to the logs to know that they've made an error with their content.

To create a circular reference. Add the following to signup.pug:

<Include>./landing-page/call-to-action.md</Include>

As you try to refresh the landing page, you'll never see the browser finish loading the page. You'll need to stop the server as well, as it registers a new callback with each cycle, so you'll never be able to handle other processes.

So let's keep track of the previously seen templates. Let's do so with an immutable list structure.

interface List<T> {
  isEmpty() : boolean;
  push(item: T) : List<T>;
  has(item: T) : boolean;
  toArray() : T[];
}

class Empty<T> implements List<T> {
  isEmpty() : boolean { return true }
  push(item : T) : List<T> {
    return new Pair<T>(item, this);
  }
  has(item: T) : boolean { return false; }
  toArray() : T[] { return []; }
}

class Pair<T> implements List<T> {
  readonly head : T;
  readonly tail : List<T>;
  constructor(head: T, tail : List<T>) {
    this.head = head;
    this.tail = tail;
  }
  isEmpty() : boolean { return false; }

  push(item : T) : List<T> {
    return new Pair<T>(item, this);
  }

  has(item : T) : boolean {
    let current : List<T> = this;
    while (current instanceof Pair) {
      if (current.head === item)
        return true;
      current = current.tail;
    }
    return false;
  }
  toArray() : T[] {
    let result : T[] = [];
    let current : List<T> = this;
    while (current instanceof Pair) {
      result.push(current.head);
      current = current.tail;
    }
    return result;
  }
}

Then we just need to create the list element as part of the template traversal. If the current template has been previously seen, instead of continuing, we return an error message.

We first refactor render as follows:

  render(templateFilePath : string, locals ?: {[key: string]: any}) : Promise<string> {
    return this.renderNoCycle(templateFilePath, locals || {}, new Empty<string>());
  }

The bulk of the render code is pulled out and put into renderNoCycle:

  renderNoCycle(templateFilePath : string, locals: {[key: string]: any}, prev : List<string>) : Promise<string> {
    let fullTemplateFilePath = this.getFullTemplatePath(templateFilePath);
    if (prev.has(fullTemplateFilePath)) {
      return this.renderCycleError(fullTemplateFilePath, prev);
    }
    return this.getTemplate(templateFilePath)
      .then((template) => {
        let result = template.fn(locals);
        let [ fragments , includePaths ] = this.splitByIncludes(result);
        let output : string[] = [ fragments[0] ];
        let fullIncludePaths = includePaths.map((includePath) => {
          return this.getRelativeFullPath(fullTemplateFilePath, includePath);
        });
        return Promise.map(fullIncludePaths, (includePath) => {
          return this.renderNoCycle(includePath, locals, prev.push(fullTemplateFilePath));
        })
          .then((files) => {
            for (var i = 0; i < files.length; ++i) {
              output.push(files[i]);
              output.push(fragments[i + 1]);
            }
            return output.join('');
          })
      })
  }

Then we finally implement renderCycleError as:

  renderCycleError(fullTemplateFilePath : string, prev : List<string>) : Promise<string> {
    return Promise.resolve<string>(`<div><h1>Cycle error</h1><p>${fullTemplateFilePath}</div>`);
  }

With this change we can see the landing page now loads successfully with the error showing.

We can further create a cycle error template and render the result there instead.

// _cycle-error.pug
.error
  h1 Content Cycle Error
  p1 Cycle detected at #{filePath}
  ul
    p Stack
    for item in stack
      li
        p #{item}

And we modify renderCycleError to load this file instead:

  renderCycleError(fullTemplateFilePath : string, prev : List<string>) : Promise<string> {
    return this.render('_cycle-error.pug', {
      filePath: fullTemplateFilePath,
      stack: prev.toArray()
    });
  }

Conclusion

By adding the capability to refresh templates as the content are modified, as well as disallowing circular template references, our Template Engine is now much more robust and user friendly, and can be used as the basis for heavy duty content management, including creating arbitrary pages like the landing page. We'll continue to build out the remaining portion of the landing page in subsequent sessions.