Session 8 - Landing Page Part 3 - Adding "Include" To Template Processor

Goal

In Session 7, we have looked at two methods of including content into the template:

Both of the methods work but are limited if you want to give maximum flexibility to content author. To do so, let's take a look at a third method, which is to create an include directive.

What Is An Include Directive

An include directive is part of the template language that would insert the content of the included file into the original file.

For example, Pug has its own include directive:

doctype html
html
  include includes/head.pug
  body
    h1 My Site
    p Welcome to my super lame site.
    include includes/foot.pug

Include Markdown Inside Pug

It even work with including Markdown files with the use of a filter.

First, we need to add a custom filter to Pug as follows:

var md = new MarkdownIt({ html: true });
require('pug').filters['md'] = function (data : string) {
  return md.render(data);
};

Then we can have a include like:

include:md <path to the markdown file>

We can then convert our index.pug to as follows:

doctype
html
  head
    title Index page
    link(rel='stylesheet', href='/css/main.css')
    // for mobile
    meta(name='viewport', content='width=device-width, initial-scale=1')/
  body
    // header section
    nav.top-nav
      div.container-fluid
        div.navbar-header
          a.navbar-brand(href='/') Header Section
    // body section
    div.container-fluid
      div.jumbotron 
        include:md ../static/landing-page/jumbotron.md
        div.sign-up Signup
      div.proposition 
        include:md ../static/landing-page/proposition.md
      div.benefits
        div.benefit
          include:md ../static/landing-page/benefits/benefit-1.md
        div.benefit
          include:md ../static/landing-page/benefits/benefit-2.md
        div.benefit
          include:md ../static/landing-page/benefits/benefit-3.md
      div.topics
        div.topic
          include:md ../static/landing-page/topics/topic-1.md
        div.topic
          include:md ../static/landing-page/topics/topic-2.md
        div.topic
          include:md ../static/landing-page/topics/topic-3.md
      div.call-to-action 
        include:md ../static/landing-page/call-to-action.md
        div.sign-up Signup
    // footer section
    div.footer Footer section

Notice that instead of using for to loop over the list of topics and benefits anymore. If we tried to continue using for as follows:

for topic in ['topic-1.md', 'topic-2.md', 'topic-3.md']
  div.topic
    include:md ../static/landing-page/#{topic}

We get the following error:

Error: ENOENT: no such file or directory, open '.../module-core/static/landing-page/#{topic}.pug'

That means the evaluation of include in Pug happens before the variable topic gets interpolated. If we want to overcome this issue, we cannot rely on the built-in include mechanism anymore due to the order of execution. Instead, we'll have to gain control over the rendering process by delaying the rendering of the markdown files until after the variables have been substituted. The easiest way is return custom elements that were generated by the loop. I.e. if we return an element called Include (notice that it's uppercase I; Pug's include is built-in and cannot be treated as an element):

for topic in ['topic-1.md', 'topic-2.md', 'topic-3.md']
  div.topic
    Include ../static/landing-page/#{topic}

The above would get transformed into

<Include>../static/landing-page/topic-1.md</Include>
<Include>../static/landing-page/topic-2.md</Include>
<Include>../static/landing-page/topic-3.md</Include>

Then if we could intercept the generated HTML before it's sent out as the response, we'll have a chance to process the elements and transform them into the Markdown content, before sending out the response.

Unfortunately, to go down this path it means that we wouldn't be able to directly make use of the Pug's built-in Express integration, since we would need to intercept the results. It means we need to make use of Pug directly. What we give up in convenience though, we can make it back with the gain in power and flexibility. Let's see how we could do that.

Custom Handling Our Own Pug Template Generation.

The way to use Pug directly is via its JavaScript API:

import * as pug from 'pug';
let fn = pug.compileFile(<path to pug file>, <pug options>);
fn(<variables for substitutions>); // generate the html result.

You might have noticed that Pug's IO API is synchronous, unlike what we have been discussing as being the norm in the Node.js' world. It's an interesting artifact that Pug has such a prominent position, given that how much emphasis that the rest of the Node.js is purely asynchronous.

With the above, we can replace res.render('index', { ... }) with the following:

import * as pug from 'pug';
...

function render(templateFilePath: string, locals ?: {[key: string]: any}) : string {
  let fn = pug.compileFile(templateFilePath, {
    filename: templateFilePath,
    basedir: __dirname
  });
  return fn(locals);
}

Be sure to add Pug's type definition via npm install --save-dev @types/pug.

The current res.render call can then be replaced with the following:

app.get('/', function (req, res, next) {
  return res.end(render(path.join(__dirname, 'views', 'index.pug'), {
  }))
});

Notice that we have removed the call to readMarkDownFileList as we are now leveraging Pug's built-in include mechanism at this time.

We can go a bit further by caching the compiled template function so we don't have to re-compile every time with the following:

let templateMap : {[key: string] : (locals ?: {[key: string] : any}) => string} = {};

function getTemplate(templateFilePath : string) {
  if (!templateMap[templateFilePath]) {
    let fn = pug.compileFile(templateFilePath, {
      filename: templateFilePath,
      basedir: __dirname
    });
    templateMap[templateFilePath] = fn;
  }
  return templateMap[templateFilePath];
}

function render(templateFilePath: string, locals ?: {[key: string]: any}) : string {
  let fn = getTemplate(templateFilePath);
  return fn(locals);
}

We can go even a bit further by allowing the passing in of a relative path against a base path (like the current __dirname) as follows:

function getTemplate(templateFilePath : string, basePath : string = __dirname) {
  let fullTemplateFilePath = path.join(basePath, templateFilePath);
  if (!templateMap[fullTemplateFilePath]) {
    let fn = pug.compileFile(fullTemplateFilePath, {
      filename: path.basename(fullTemplateFilePath),
      basedir: basePath
    });
    templateMap[fullTemplateFilePath] = fn;
  }
  return templateMap[fullTemplateFilePath];
}

Then the rendering call can be changed to:

return res.end(render('views/index.pug'));

If we allow the passing in of a different basePath it could even get rid of the views directory. But since we are trying to keep render looking similar to res.render we would need a different approach - having a TemplateEngine object that would hold the basePath.

Let's rewrite our render-related code into a class called TemplateEngine:

class TemplateEngine {
  private _templateMap : {[key: string] : (locals ?: {[key: string] : any}) => string};
  private _basePath : string;

  constructor(basePath : string) {
    this._basePath = basePath;
    this._templateMap = {};
  }

  getTemplate(templateFilePath : string) {
    let fullTemplateFilePath = path.join(this._basePath, templateFilePath);
    if (!this._templateMap[fullTemplateFilePath]) {
      let fn = pug.compileFile(fullTemplateFilePath, {
        filename: path.basename(fullTemplateFilePath),
        basedir: this._basePath
      });
      this._templateMap[fullTemplateFilePath] = fn;
    }
    return this._templateMap[fullTemplateFilePath];
  }

  render(templateFilePath : string, locals ?: {[key: string]: any}) : string {
    let fn = this.getTemplate(templateFilePath);
    return fn(locals);
  }
}

let templateEngine = new TemplateEngine(path.join(__dirname, 'views'));

...

app.get('/', function (req, res, next) {
  return res.end(templateEngine.render('index.pug'));
});

Loading Markdown Files Via Processing HTML Elements

Now that we have control over the rendering process, the next step is to process the Include elements and transform them into the corresponding Markdown content.

There are a couple of ways to deal with the transformation:

  1. Use regular expression to search and replace.
  2. Convert the HTML into its DOM representation, search the DOM for replacement, finally serialize the DOM back out to string.

The second method is more general, however, it is also more effort to implement, so for now we'll make use of the regular expression's approach, until we come across a need that cannot be easily met with regular expressions.

Let's try to search our HTML for the Include element with the following pattern:

  let includeRegex = /<Include>\s*([^<\s]+)\s*<\/Include>/g;

  ...
  splitByIncludes(html : string) : [ string[] , string[]] {
    let split = html.split(includeRegex);
    let fragments : string[] = [ split[0] ];
    let includePaths : string[] = [];
    for (var i = 1; i < split.length; i = i + 2) {
      includePaths.push(split[i]);
      fragments.push(split[i + 1]);
    }
    return [ fragments, includePaths ];
  }

splitByIncludes splits the html into fragments, interleaved by the path value inside the Include element. We can then pull out the include path from the fragments, transform the include path into the corresponding content, and finally reconstruct them by combining the values.

Then we can modify our render function to get the list of the path, and then based on the list of the path, process the included files. Keep in mind that reading files should be kept an async operation in Node.js (even though Pug went against the convention with its API).

We'll rewrite our render as follows:

  getFullTemplatePath(templateFilePath : string) : string {
    return path.join(this._basePath, templateFilePath);
  }

  getRelativeFullPath(fullFilePath : string, relativePath : string) : string {
    return path.join(path.dirname(fullFilePath), relativePath);
  }

  render(templateFilePath : string, locals ?: {[key: string]: any}) : Promise<string> {
    let fullTemplateFilePath = this.getFullTemplatePath(templateFilePath);
    let fn = this.getTemplate(templateFilePath);
    let result = fn(locals);
    let [ fragments , includePaths ] = this.splitByIncludes(result);
    let output = [ fragments[0] ];
    let fullIncludePaths = includePaths.map((includePath) => {
      return this.getRelativeFullPath(fullTemplateFilePath, includePath);
    });
    return Promise.map(fullIncludePaths, (includePath) => {
      return this.render(includePath, locals);
    })
      .then((files) => {
        for (var i = 0; i < files.length; ++i) {
          output.push(files[i]);
          output.push(fragments[i + 1]);
        }
        return output.join('');
      })
  }

Notice that render now returns a Promise<string> instead of just a string, so we'll need to modify our caller as:

app.get('/', function (req, res, next) {
  templateEngine.render('index.pug')
    .then(result => res.end(result))
    .catch(next);
});

The above gives us the structure necessary to handle the Include element during runtime, but our TemplateEngine doesn't know how to render a Markdown file yet! So let's fix that as well by integrating the Markdown functions into TemplateEngine as well - this will give us the ability to have a Markdown-based template!

Let's first pull out the type definition of the template:

type Template = (locals ?: {[key: string] : any}) => string;

class TemplateEngine {
  private _templateMap : {[key: string] : Template};
  ...
}

Then let's pull out the compilation function as follows:

  compileTemplate(fullTemplateFilePath : string) : Promise<Template> {
    let extname = path.extname(fullTemplateFilePath);
    switch (extname) {
      case '.pug':
        return Promise.try<Template>(() => pug.compileFile(fullTemplateFilePath, {
          filename: path.basename(fullTemplateFilePath),
          basedir: this._basePath
        }));
      case '.md':
        return fs.readFileAsync(fullTemplateFilePath, 'utf8')
          .then((data) => {
            return (locals : any) => {
              return md.render(data, { html: true });
              };
          })
      default:
        throw new Error(`UnknownTemplateType: ${extname}`);
    }
  }

Since compileTemplate is asynchronous, getTemplate will also need to become asynchronous:

  getTemplate(templateFilePath : string) : Promise<Template> {
    let fullTemplateFilePath = this.getFullTemplatePath(templateFilePath);
    if (this._templateMap[fullTemplateFilePath]) {
      return Promise.resolve<Template>(this._templateMap[fullTemplateFilePath]);
    } else {
      return this.compileTemplate(fullTemplateFilePath)
        .then((template) => {
          this._templateMap[fullTemplateFilePath] = template;
          return template;
        })
    }
  }

And its caller also need to become asynchronous:

  render(templateFilePath : string, locals ?: {[key: string]: any}) : Promise<string> {
    let fullTemplateFilePath = this.getFullTemplatePath(templateFilePath);
    return this.getTemplate(templateFilePath)
      .then((fn) => {
        let result = fn(locals);
        let fullIncludePaths = this.findIncludePaths(result).map((includePath) => {
          return this.getRelativeFullPath(fullTemplateFilePath, includePath);
        });
        return Promise.map(fullIncludePaths, (includePath) => {
          return this.render(includePath, locals);
        })
          .then((files) => {
            let fragments = this.splitByIncludes(result);
            let output = [ fragments[0] ];
            for (var i = 0; i < files.length; ++i) {
              output.push(files[i]);
              output.push(fragments[i + 1]);
            }
            return output.join('');
          })
      })
  }

With this change, we can write our index.pug as follows:

doctype
html
  head
    title Index page
    link(rel='stylesheet', href='/css/main.css')
    // for mobile
    meta(name='viewport', content='width=device-width, initial-scale=1')/
  body
    // header section
    nav.top-nav
      div.container-fluid
        div.navbar-header
          a.navbar-brand(href='/') Header Section
    // body section
    div.container-fluid
      div.jumbotron 
        Include ../static/landing-page/jumbotron.md
      div.proposition 
        Include ../static/landing-page/proposition.md
      div.benefits
        for benefit in [1, 2, 3]
          div.benefit
            Include ../static/landing-page/benefits/benefit-#{benefit}.md
      div.topics
        for topic in [1, 2, 3]
          div.topic
            Include ../static/landing-page/topics/topic-#{topic}.md
      div.call-to-action 
        Include ../static/landing-page/call-to-action.md
    // footer section
    div.footer Footer section

What is really cool now is that we can use Include in Markdown files as well. Let's move div.signup Signup from the template to jumbotron.md and call-to-action.md.

// jumbotron.md
# This is the Jumbotron Section

This is the Jumbotron Section.

<Include>../../views/signup.pug</Include>
// call-to-action.md
## Call To Action

This is the Call To Action Section.

<Include>../../views/signup.pug</Include>
// signup.pug
div.signup This is the Signup Section.

And our system can now handle both Pug and Markdown as template files, with the ability to include other types of template files from either.

Conclusion

By moving to a custom TemplateEngine, we now have given content authors a powerful capability to create contents dynamically without having to modify the source code. We'll next look at a couple of issues that we need to address with our Template Engine in the next session.