Module Core - Session 3 - Serve Static and Dynamic Files

Welcome to Session 3. We will continue to push our small web server toward our dream app.

Goal

In Session 2, we covered the following:

Knowing how to route give us the blueprint on how to structure the rest of the web application. We will continue to examine how to handle other common routing scenarios:

Serving Static Files

We have actually see a basic version of this already - in Session 1 we have been reading the read.js and serving it out as a plain text file, however, we are only serving a single file no matter what URL paths was specified, so we need to change that to the following:

  1. Identify the particular file that user is requesting for based on the URL
  2. Locate the file, read its content and serve it.
  3. Handling errors as necessary (file not found, for example).

Let's make that a bit more concrete, let's say that I have a folder called static, and it has a couple of files:

And I want to map the URL /index.html to static/index.html, and then /test.html to static/test.html, in a general pattern being that anything under / should be mapped to the static folder.

The structure of the handler would then look like this.

// NOTE that I'm using next in a handler to handle async errors that would be lost otherwise.
function staticFileHandler(req, res, next) { 
  var mappedPath = mapUrlToFilePath(req.url); // translates the url to underlying file path.
  fs.readFile(mappedPath, 'utf8', function (err, data) {
    if (err) {
      next(err);
    } else {
      res.end(data);
    }
  });
}

Then we just need to define the mapUrlToFiePath function to do what we want. For now, we assume that the static folder is inside the project folder, so we can hardcode that with __dirname.

function mapUrlToFilePath(url) {
  // as index.js is at project root, this translates to <project root>/static/<url>
  return path.join(__dirname, 'static', url);
}

Then we just need to include it in the express routes.

app.get(staticFileHandler);

Restart and test that /index.html and /test.html returns you the desired files, but /bar.html will throw an error as there is no bar.html in the static folder.

File Encoding

We should test that with other types of files as well, like for example, images.

Create a test.png file and put it into the static folder, then try to retrieve it via /test.png. It turns out that in this case we would end up with a corrupted image that we cannot open (try it - your image viewer will complain that it's a corrupted file).

The reason it's corrupted is because that when we read the file content via fs.readFile, we have passed in the encoding of utf8, which will treat the read-in file as text. That works for text files, but doesn't work for images, which is just a binary file. So let's remove the encoding.

  fs.readFile(mappedPath, function (err, data) {

When fs.readFile is called without the encoding, it returns a Buffer object instead of a string, and the Buffer object simply holds the binary data as is. Luckily we don't have to change our output function, as res.end can take either a string or Buffer.

Restart the server and now you'll see /test.png retrieves a valid image.

How express Serves Static Files

As stated before, serving static file is a very common use case, so express has this particular functionality via express.static like below:

app.use(express.static(<the root path of the static file folder>))

I.e, with our existing root path of the static folder, it would be:

app.use(express.static(path.join(__dirname, 'static')));

Which is the equivalent of what we have done, except that express.static has a different error handler for non-existent files.

Serving Out Markdown Files

Since raw html files are cumbersome to write, people have invented a few different ways of creating the HTML files, some of which were WYSIWYG-based, while others are text-based.

WYSIWYG-based means that we need to create a WYSIWYG editor for editing the HTML, that's a more involved process. So let's first address the simpler process, which is to have a text-based markup file that can get converted over to HTML. Markdown has emerged as the go-to format for such choices.

Therefore, besides serving out raw static files, we also want to serve out the markdown files. But we will not do it in the format that we would write in, but in HTML format after it's been converted.

Let's tackle this step by step. Before we try to serve out the dynamically generated HTML, let's just first be sure that we could serve out the raw markdown files.

Let's create a couple of markdown files in a different directory, call it content.

Then let's add another express.static route that points to the content directory.

app.use(express.static(path.join(__dirname, 'content')));

As stated, express routes are prioritized by the order, and the first one wins, so put it to where you believe it should go in terms of priority.

After restarting the app you'll be able to point to /test.md and /test2.md in your browser and see the raw markdown files that you have added.

Moving Toward Dynamic Content Generation

Unfortunately, express.static doesn't give us a hook to transform the files while it's being served, so we'll have to go back to make use of our raw static file handler to do our own file handling.

The first thing we need to do is to bring it to par, since we have previously hardcoded the static path into the functions, so we must refactor it in order for us to pass in the content directory.

function mapUrlToFilePath(url, folderName) {
  return path.join(__dirname, folderName, url); // refactor static into a parameter.
}

Then we must change our handler to a handler factory that takes in the folderName variable and creates the handler.

function configurableStaticFileHandler(folderName) {
  return function staticFileHandler(req, res, next) {
    var mappedPath = mapUrlToFilePath(req.url, folderName);
    fs.readFile(mappedPath, function (err, data) {
      if (err) {
        next(err);
      } else {
        res.end(data)
      }
    })
  }
}

And change the actual route instantiation to:

app.use(configurableStaticFileHandler('content'));

Restart the web server to verify that we have fully replaced express.static call with our own configurableStaticFileHandler.

The last step then is to process the data parameter before it gets returned via res.end. This is the place where we can transform the markdown into HTML.

// something like this...
res.end(markdownToHtml(data))

Let's find ourselves a markdown parser and transformer - as a popular format there are already plenty third-party libraries we could use. We'll use the one called markdown-it (you can use other ones if you choose to, there are no harms to try multiple libraries to find the one that you like).

npm install --save markdown-it

Then load the library:

var MarkdownIt = require('markdown-it');

Then make use of it based on its documentation, and change the encoding back to utf8 since we are now dealing with text:

function configurableStaticFileHandler(folderName) {
  return function staticFileHandler(req, res, next) {
    var mappedPath = mapUrlToFilePath(req.url, folderName);
    let md = new MarkdownIt();
    fs.readFile(mappedPath, 'utf8', function (err, data) {
      if (err) {
        next(err);
      } else {
        res.end(md.render(data))
      }
    })
  }
}

Save and restart the web server and browse to /test.md, voila, we are now seeing the generated HTML file!

Handling Both Text & Binary Files

Since we change the encoding back to utf8, we are encountering the same issue as before that this handler could only serve text files. Perhaps in the future we might want to consider handling binary files and text files through this handler as well. To do so, let's remove utf8 again, but we now need to determine if the file is a text file or binary file, so we can decide how we need to process it.

A simple way to determine the processing rule is via the file extension, such as:

With that we can rewrite our data handling branch as follows:

    ...
    fs.readFile(mappedPath, 'utf8', function (err, data) {
      if (err) {
        next(err);
      } else {
        switch (path.extname(data)) {
          case '.md':
            try {
              return res.end(md.render(data.toString('utf8'))) // use Buffer's toString('utf8') to get the text.
            } catch (e) { // render could fail synchronously, so try / catch 
              return next(e);
            }
          default:
            return res.end(data);
        }
      }
    })
  }
}

With this new function, we could actually combine the static folder, which holds the HTML and image files, with the content folder, which holds the markdown files. And we could also consolidate the two routes into one.

Although, keep in mind that if we only keep our static handler route, it does not allow serving out of the raw markdown files anymore. We might need to add it back in in the future when we need to serve out the raw markdown files.

Conclusion

With the ability to serve out raw files and generated files, we basically have a very primitive CMS (Content Management System) in our hands. We can write out the files in either Markdown or HTML, and then serve them out with our web server. Although the pages would be really ugly at this time, and there are still other functionalities that we want (for example, allow us to write file snippets and combine them together), they can be built on top of what we have done so far, and we just need to keep enhance our system to get there.

Let's continue in the next session.

App From Scratch, LLC. Copyright 2017.