Revisiting images formats for the web

Every so often I see comparisons between image formats that say one format is better than others or that one format is better for a given task but I’ve always wondered where the numbers came from and what testing criteria were used. Rather than take things at face value, I want to make sure that whatever decision I make it is backed up with data. I’ve put all the files and scripts on a Github repository for you to run the same tests and see if the results match mine. Beware that the TIFF images are very large and may take a large chunk of your data plan if you download them on mobile.

This post does not cover HEIC/HEIF and AVIF image formats. To cover those two formats well, I need more time to compile and test the tools and want to make sure that I don’t mix tool research with image quality. I will post the results of any research on those two formats in a separate post.

Before we jump into looking at the quality of compressed images, let’s take a look at what’s available and what’s coming down the pipeline as far as image formats for the web.

Format Initial Release Open? Type Available Encoder Encoder to use Notes
GIF 1987 No Lossless Yes ImageMagick
JPG 1992 Yes Lossy Yes ImageMagick According to Wikipedia patents for JPEG technologies expired
PNG 1996 Yes Lossless Yes ImageMagick Also Provides animation support. Check [caniuse.com](https://caniuse.com/#feat=apng) for supported browsers
WebP 2010 Yes Lossless and Lossy Yes ImageMagick or directly with cwebp Based of WebM video compression

The process

As Kornel Lesinski writes in How to compare images fairly:

Absolutely the worst way to compare images is to convert one lossy format to another and conclude you “can’t see the difference”.

Why is it bad? Save a photo as a couple JPEGs at quality=98 and quality=92. It will be hard to tell them apart, but their file sizes will differ by nearly 40%! Does it prove that JPEG is 40% better than… itself? No, it shows that “quality appears the same, but the file much is smaller!” can easily be nonsense that proves nothing.

To make a fair comparison you really have to pay meticulous attention to encoder settings, normalizing quality, and ensuring that compared images are in fact comparable.

It’s really hard to make a fair comparison.

Before we even start encoding the images we have to do a few things:

  • Find a lossless, high-quality image to use as the source for the exercises
  • Decide what tools we will use to encode the images. Sometimes this may be decided for you as there may not be many tools for the newer formats
  • Decide what criteria you will use for your testing and how will you measure it
  • Will you measure objective quality using tools like DSSIM?
  • Will you compare file size for a given quality?
  • How will you decide which metric is more important?
  • Make sure that the tools produce similar output. For example, all formats should use chroma subsampling or none should
  • Figure out what are the equivalent settings for the formats that you’re testing. Q=80 for a JPEG image may not be the same as Q=80 for other formats
  • Test all formats at the same or equivalent quality
  • Make sure that all format encoders can work from the same source
  • If the format offers lossless and lossy compression use lossy to match what JPEG does

I chose to work with different images in TIFF format. Information about the specific images is listed below:

We’ll use different encoders for different formats, below is the list of formats with their associated image encoders. All the binaries were reinstalled to make sure I have the latest versions available via Homebrew as of this writing:

  • PNG: ImageMagick
  • JPG: ImageMagick
  • WebP: cwebp

First Test: Equal Quality to measure file size

The first test I wanted to run is what happens if we encode a TIFF source image to all image formats that we can test with the same quality value, in this case, 80. It is important to note that a JPEG image encoded at 80 quality is not the same as a lossy WebP image encoded at the same quality. We do it this way because it’s the easiest way to test and it’s what I would do in Photoshop of when running image compression with tools like imagemin

The questions that I want to answer with this test:

Keeping quality constant, what lossless format provides the smaller file size?

Rather than type the command every time that I run the test, and to make the results reproducible, I created the Bash script below

#! /usr/bin/env bash

# Variable holding name of source image.
SOURCE_IMAGE='STSCI-H-p2022a-f-4398x3982'

# Variables holding names of
# encoders' binaries
IMAGE_MAGICK='convert'
WEBP_ENCODER='cwebp'
HEIC_ENCODER='heif-enc'

echo Starting First Encoding Test

if hash ${IMAGE_MAGICK} 2>/dev/null; then
  echo encoding to PNG
  ${IMAGE_MAGICK} ${SOURCE_IMAGE}.tif \
  -quality 80 ${SOURCE_IMAGE}.png
  echo encoding to JPG
  ${IMAGE_MAGICK} ${SOURCE_IMAGE}.tif \
  -quality 80 ${SOURCE_IMAGE}.jpg
else
  echo cannot convert to PNG or JPG
fi

if hash ${WEBP_ENCODER} 2>/dev/null; then
  echo encoding to lossy WebP
  ${WEBP_ENCODER} -q 80 \
  ${SOURCE_IMAGE}.tif \
  -o ${SOURCE_IMAGE}.webp
else
  echo cannot convert to WEBP
fi

if hash ${HEIC_ENCODER} 2>/dev/null; then
  echo encoding to lossy HEIC
  ${HEIC_ENCODER} --quality 80 \
  ${SOURCE_IMAGE}.png
else
  echo could not encode to HEIC
fi

My Results with images encoded from TIF high-quality sources and JPG where TIF was not an option:

Format File Size
TIFF (base) 15MB
PNG 13.9MB
JPG 855KB
WebP 266KB

There is a lot of research and tweaking to obtain optimal results.

So in the naïve, all quality is the same test, WebP wins by a lot. remember that image quality is not a straight equivalency across formats as explained earlier.

Finding the optimal quality

I know that optimal quality depends on the type of image and the screens we’re working with, but an initial step on determining our optimal quality may be to establish what are the best compression settings for each format. We’re likely to be serving at least two with our sources or srcset images.
To answer this question we’ll do a two-step process:

  • We create a set of WebP and a set of JPG images with quality ranging from 50 to 100
  • We’ll use SSIM to provide an objective metric to use in comparing the images.

We then analyze the SSIM results to decide which of the compressed images gives us the best combination of quality measured by SSIM against a 100 quality compressed PNG image (for some reason ImageMagick’s compare command will not work against a TIFF source image) and file size.

The Scripts

Each step of the process uses its own script.

The first scripts generate multiple images. It is essentially the same script that we used for the previous evaluation, except that we have to pass the name of the image, without extension, as a parameter when we invoke the script.

if hash ${WEBP_ENCODER} 2>/dev/null; then
  for i in {50..100..10}
    do
      echo encoding to lossy WebP at ${i} quality
      ${WEBP_ENCODER} -q ${i} \
      ${SOURCE_IMAGE}.tif \
      -o ${SOURCE_IMAGE}-${i}.webp
  done
else
  echo cannot convert to WEBP
fi

The second script does the comparison. I would much rather use a direct SSIM encoder rather than using ImageMagick’s compare command but none of the available encoders works as well as I would like.

So we use magick compare as our comparison tool and we run it against every WebP and JPG image we created using the 100 quality PNG as our standard to compare against.

I’d rather compare against the TIFF image but IM errors out with that comparison so, in my opinion, PNG is the next best available testing option.

This code will only work on Bash shells version 4 and higher.

SOURCE_IMAGE=$1
IMAGE_MAGICK_COMPARE='magick compare'

echo starting to work with SSIM comparison

if [ -f ${SOURCE_IMAGE}-100.png ]; then
  echo ${SOURCE_IMAGE}-100.png exits
  for i in {50..100..10}
    do
      echo "running comparisons for webp at ${i} quality"
      ${IMAGE_MAGICK_COMPARE} -metric ssim \
      ${SOURCE_IMAGE}-100.png \
      ${SOURCE_IMAGE}-${i}.webp \
      null:
  done

  for i in {50..100..10}
    do
      echo "running comparisons for jpg at ${i} quality"
      ${IMAGE_MAGICK_COMPARE} -metric ssim \
      ${SOURCE_IMAGE}-100.png \
      ${SOURCE_IMAGE}-${i}.jpg \
      null:
  done
else
  echo can\'t run the WebP comparison
fi

The Results

The first table uses STSCI-H-p2022a-f-4398×3982 from the Nasa image library. The image is 14MB and may use a significant portion of your data plan on mobile.

The file sizes are generally what I expected and the differences between the SSIM values are similar enough to make file size becomes the primary consideration.

Quality WebP File Size WebP SSIM Value JPG File Size JPG SSIM
100 2.1MB 0.986584 10.9M 0.992733
90 639KB 0.981029 3.6MB 0.985442
80 266KB 0.975904 2.1MB 0.982087
70 183KB 0.973957 1.5MB 0.978859
60 153KB 0.973044 1.1MB 0.974777
50 128KB 0.972133 864KB 0.972219

The second table uses geisha-high-resas the image to test. This is a much brighter and deep contrast color image. The image is 11.9MB and may use a significant portion of your data plan on mobile.

Quality WebP File Size WebP SSIM Value JPG File Size JPG SSIM
100 2.6MB 0.980442 5.2MB 0.992432
90 757KB 0.961136 1.4MB 0.966931
80 300KB 0.947507 788KB 0.956424
70 217KB 0.943466 535KB 0.949292
60 188KB 0.941836 407KB 0.942214
50 166KB 0.939598 332KB 0.937974

The final example, the USS California image presents some interesting variance for analysis. The image is 12.8MB and may use a significant portion of your data plan on mobile.

I ran the compression and the SSIM comparison in separate steps, so I made the incorrect assumption that grayscale WebP images would exhibit the same behavior as color ones where the WebP files scored better in the SSIM metric across the board instead of fluctuating as they did. Need to do more research, particularly if it has to do with encoding settings on the WebP size and whether the -jpeg_like and -shap_yuv flags would change the results in any way.

Quality WebP File Size WebP SSIM Value JPG File Size JPG SSIM
100 10.1MB 1 11.2MB 0.998955
90 3MB 0.956338 3.6MB 0.951193
80 1.1MB 0.909765 2.1MB 0.930837
70 669KB 0.891505 1.5MB 0.91801
60 530KB 0.88458 1.1MB 0.907007
50 426KB 0.877016 864MB 0.898054

What’s missing

Two additional image formats should be in the encoding tests but are not.

HEIC is an image format based on the HEVC video code. The only encoder I found for the format produced significantly larger sizes than any other formats. I need to run additional tests to make sure I’m encoding it as it should be and that the larger sizes are not a result of my doing it wrong.

AVIF is the image format based on the open-source AV1 video format. The same encoder that creates HEIC files (lib-heif) will, supposedly, create AVIF files when running with a flag. I have yet to get it to work.

I was able to install the libavif reference implementation but I’ll take a little time to document the process and then set up another test to see how heic and avif compare to WebP, JPG, and PNG.

Conclusions

There is additional work that needs to happen regarding format-specific configuration and encoding flags and test them to see if the formats are affected by the flags you use when doing the compression.

Depending on the type of images you use on your site or project, evaluating the file sizes of a representative number of images is always a good idea.

Thanks to Jeremy Wagner for proofreading and giving technical feedback on the post.

Customizing WordPress

How do we customize WordPress? How do we make themes our own and add functionality that is not part of a theme or that you want to use regardless of the theme you have installed?

This post will explore the basics of creating custom WordPress elements and will cover plugins, child themes, and custom themes.

Note

The specifics of your situation will be different than mine. You should always analyze your needs and requirements before making a decision.

Planning

The first thing to do when planning to add things to a theme is to decide what’s the best alternative to get it done.

Do you use an existing plugin to work with your theme? WordPress has a huge collection of plugins to accomplish a variety of tasks; it may just pay to evaluate existing plugins to see if they work.

Same things with plugins. Evaluate existing plugins and test the existing capabilities and expand from there. Only if you evaluate existing themes and none do what you want without extensive modifications you should consider building a brand new theme using a starter kit like _s or the themes listed in articles like 10 Best WordPress Starter Themes for Developers in 2020 or 19 Best WordPress Starter Themes for Developers in 2020. Test the themes and don’t take anything for granted.

Plugins

I recommend using plugins when you have a single-purpose task or script that you want to use across multiple themes or sites.

For example, in a previous post I discussed how to add the scripts for Quotebacks to my WordPress installation. I want to make sure that this functionality works on every theme I may want to use so, rather than copying the enqueue function to whatever theme I choose to use, I create a simple plugin to do it once and leave it in the site regardless of the theme

I won’t go into the details of creating a plugin, we could spend entire articles covering just that. Instead, I will refer you to Plugin Basics from the Plugin Developer Handbook and WordPress Essentials: How To Create A WordPress Plugin as good starting points to use when creating your own plugins.

Themes

Themes are the look and feel of your WordPress site. They can also bundle functionality that the developer considered essential for the theme to work although the bundling is not required, you can give users a list of plugins to install on their own.

We’ll discuss two ways to work with WordPress themes: Child Themes and Custom themes from scratch.

Child Themes

Child themes are the easiest way to customize a theme without writing them from scratch. Some of the advantages of using child themes:

  • You’re building on something that already exists, thus speeding up development time
  • You can leverage the functionality of sophisticated frameworks and parent themes while customizing the design to your needs
  • Upgrade the parent theme without losing your customizations
  • Since you didn’t modify the parent theme you can disable the child theme and everything will be as it was before
  • It’s a good way to learn

Assuming that you’ve already uploaded the theme that you want to use as a parent; the idea is to create an empty folder and add two required files.

To create a child theme follow these steps:

  1. Create a folder for your theme (in this case we’ll call it Twenty Twenty Rivendellweb)
  2. In the folder create the following files
    • a style.css stylesheet
    • a functions.php file

For now, don’t worry that the files are empty. We’ll populate them as we move forward in the tutorial.

In styles.css add the following comment.

/*
 Theme Name:   Twenty Twenty Rivendellweb
 Theme URI:    https://github.com/caraya/2020-child
 Description:  Twenty Twenty Child Theme
 Author:       Carlos Araya
 Author URI:   https://publishing-project.rivendellweb.net/
 Template:     twentytwenty
 Version:      1.0.0
 License:      GPL2.0 or later
 License URI:  https://opensource.org/licenses/GPL-2.0
 Tags:         light, dark, two-columns, right-sidebar, responsive-layout, accessibility-ready
 Text Domain:  twentytwentyrivendellweb
*/

You can add further CSS customizations to make the site look the way you want to but it’s not a requirement, the comment is enough.

In the PHP file, we need to add the parent theme’s stylesheet and the style that we just created.

To add the style sheets in a WordPress safe way we create a function, my_theme_enqueue_styles to put the code we want to run.

<?php
function my_theme_enqueue_styles() {

  // This is 'twentytwenty-style'
  // for the Twenty Twenty theme.
  $parent_style = 'parent-style';

  wp_enqueue_style( $parent_style,
    get_template_directory_uri() . '/style.css' );
  wp_enqueue_style( 'child-style',
    get_stylesheet_directory_uri() . '/style.css',
    array( $parent_style ),
    wp_get_theme()->get('Version')
  );
}
add_action( 'wp_enqueue_scripts', 'my_theme_enqueue_styles' );

The first call to wp_enqueue_styles will add the parent theme’s stylesheet by using get_template_directory()to build a URI pointing to the parent theme.

The second wp_enqueue_styles will load the child theme’s stylesheet by using get_stylesheet_directory_uri() to retrieve the locaation of the child theme’s stylesheet.

The final step is to call the wp_enqueue_scripts action with the function we created as the callback.

This is the basic child theme.

If we want to modify a template we copy it from the parent theme and modify the copy in the child theme.

Same with new templates. We create them in the child theme and edit them there.

You can find more information on child themes in How To Create And Customize A WordPress Child Theme

The Twenty Twenty child theme I created using these techniques is in a Github Repo

Custom Themes

As good as child themes are there are times when the number of changes makes the child theme strategy too cumbersome.

That’s where we create a brand new theme, either completely from scratch or using a starter theme.

There are plenty of starter themes developers to choose from. 19 Best WordPress Starter Themes for Developers in 2020 lists starter themes for you to consider.

There are themes that are bundled with libraries and designed for specific workflows. These starter themes include:

I selected Underscores (without Bootstrap) for my theme development. It gives me the best combination of ready-to-use features and ease of creating tooling around the elements we create.

Custom themes are way harder to conceptualize what we want to do and how we want to do it. Because of these difficulties, it’s hard to prescribe what to do with custom themes. I would suggest starting with the following questions:

  • What type of site am I designing?
  • Do I need to add any additional templates?
  • How am I planning to modify the existing templates?
  • Who is the audience?
  • What sites am I competing with?
  • Do I have analytics to confirm my audience? Is there a comparable site to evaluate?
  • What browsers do I have to support
    • How does browser support limit the technologies I want to use?
  • How am I handling responsive design?

These are some basic questions, there are others that may become relevant as you build the site.

I learned a lot about Underscores from Morten Rand-Hendriksen’s WordPress: Building Themes from Scratch Using Underscores course on Linkedin Learning. Although the course is from 2017, the code hasn’t changed enough, in my opinion, to not use it.

Do you know where your third parties are going?

Third-party scripts on your site present a potentially dangerous side effect. We don’t know what additional assets third-party load. That has security and performance implications. This post will look at the performance side of the issue.

As Simon Hearne writes in How to Find the Third-Parties on Your Site using third party scripts that, in turn, make calls to additional sites and parties outside their (and your) control can affect the performance of your site. Quoting the post:

I ran the homepage through WebPageTest and sure enough, there were a bunch of calls to various subdomains of facebook.com. Thankfully WebPageTest stores initiator, referer and redirect headers; so with a little work, you can find out where these third-party calls come from. The director was correct, there were no calls to Facebook on their site. It was a third-party creating fourth-party calls to Facebook! This can have serious ramifications if Facebook (or any other third-party call) affects customer experience.

So why does this matter?

As Steve Souders (Frontend SPOF and Frontend SPOF survey), Pat Meenan (Testing for Frontend SPOF), and Joshua Bixby (How vulnerable is your site to third-party failure?) write, “fourth party” scripts (scripts called from third party code) can have adverse effects on your site’s performance. You (and potentially your client) have no way of controlling what these scripts do and whether they will block or slow down rendering.

In Things to Know (and Potential Dangers) with Third-Party Scripts, Yaphi Berhanu list additional concerns about third-party scripts that go beyond performance. The article is from 2017 still presents points that are still worth researching.

Testing

A very interesting to see the number of requests your third party scripts generate is to run your site through Request Map and see the results. I was surprised to see how many of its submodules a script requested in the layout-experiments site.

The images that follow present Request Map results for three different websites, two sites I own, and cnn.com.

For the CNN request map, note that the circles are smaller and the name of individual assets is impossible to read because of the number of assets involved.

Requet map for rivendellweb.net. Created from Request Map Generator
request map for layout-experiments.firebaseapp.com
Requet map for layout-experiments.firebaseapp.com. Created from Request Map Generator
request map for cnn.com
Requet map for CNN. Created from Request Map Generator

Solving the issue

In an ideal world, you’d be able to trust your third party scripts that they will do a minimum due diligence on their dependencies.

But in the real world, we don’t have that luxury. We use what we must and we hope that scripts that are loaded by our third-party scripts will not slow down our applications.

Tools like request map will not stop the spread of unknown and unwanted scripts but they will help when clients ask you where did that script comes from and when you go to the third party script providers and ask why are the additional assets being loaded.

Implementing Quotebacks in WordPress

Quotebacks are an interesting way to cite content from other websites. They work in two stages.

Installing the extension and getting the quotations

The first one is HTML formatted with special data attributes, a footer and a link to the script. There are browser extensions to create this code, one for Chrome and one for Firefox (currently under development).

Once the extension is installed, go to a page, select text and press command + shift + S on Mac and control + shift + S on Windows

The result of highlighting text and pressing command/control + shift + S to generate the quote back. There are options to copy embed, copy markdown or close the quote at the bottom of the caption.

The final step in this section is to paste the code into the page that you want to use

Installing the script

The second part, the quoteback.js scripts converts the blockquote into a custom element with its own built-in styles. The installation is simple, either create a script and point the source to the CDN

<script
  src="https://cdn.jsdelivr.net/gh/Blogger-Peer-Review/[email protected]/quoteback.js">
</script>

Or download the script and link to it locally

<script src="js/vendor/quoteback.js></script>

Completing both steps will produce results similar to the following image.

Example of a formatted quoteback

Why Use them?

Using these types of citations keep me honest and it makes me think about the type of resources I use and how I use them. It also allows readers to get the full text and context of your citations and form their own opinions about the subject.

WordPress specifics

There are a few WordPress-specific issues that we need to address as we implement them in a WordPress site.

Linking to a CDN or hosting locally?

This is a tricky question.

If we run the plugin from CDN, we get the latest code but we add one or more request to those already being made and, potentially making the site slower since the custom elements will only be upgraded when the script finishes loading.

Keeping a local copy makes the code more efficient, it reduces the number of requests, but it puts the weight of updating the code on the plugin creator.

For this project I chose to host the script as part of the plugin.

Plugin or theme?

Whether to host it in a plugin or a theme will depend on what you want to do with it.

If you want the functionality to be part of a theme and don’t mind if the code breaks when you move to another theme, then a theme would be ok.

But if you want to keep the functionality regardless of the theme you’re using, then you must use a plugin.

For this project I’ve chosen to build it as a plugin.

Process

To create a WordPress plugin we need to add a preamble comment that contains the plugin metadata that WordPress will use to show the plugin in the admin plugin menu.

I also incorporate a basic tool to prevent direct access to the plugin. If the constant ABSPATH is not defined then the user tried to access the script directly and we don’t allow that to happen so we exit.

<?
/*
  Plugin Name: WP-Quotebacks
  Plugin URI: https://publishing-project.rivendellweb.net/
  description: Backend work for Quotebacks See <a href="https://quotebacks.net/">quotebacks.net</a>
  Version: 1.0
  Author: Carlos Araya
  Author URI: https://publishing-project.rivendellweb.net/
  License: MIT
*/

if ( ! defined( 'ABSPATH' ) ) {
  exit;
}

The main part of the script is the function and action to add the script to the pages.

We don’t add the script like we’d add it in a regular page, we need to use WordPress-specific functions, enqueue_script, to add them to all pages on the WordPress installation.

By itself enqueue_script will not add the script, for that we need to add an action that uses the enqueue_scripts hook (note the plural) and a function callback.

<?
function quotebacks_enqueue_script() {
  wp_enqueue_script('quotebacks',
  plugins_url(
    'vendor/quoteback.js', __FILE__ ),
    array(),
    '20200615',
    true
  );
}

add_action( 'wp_enqueue_scripts', 'quotebacks_enqueue_script' );

With this plugin installed we can just paste the code from the extension with one difference, we should delete the script located at the end of every embed for quotebacks.

Building a Network Navigator

In 1987 Apple released this concept video of what they called a Network Navigator, an integrated agent to manage data from different sources, communicate with people across the network, and share information regardless of the source and its location.

It may seem quaint to us now, 30 years later but the real question is whether the tools have evolved to the point where it is possible to build such technology.

This post will explore some of the current technologies that would make such a project feasible and discuss potential future implementations. This is a theoretical exploration rather than a working example. There is no code in this post 🙂

Voice Enabled

Three things that were impossible to predict in ’87 are how powerful mobile phones would become, how much would agents become a common day occurrence, and the growth of home automation with tools like Alexa Home or Google Nest.

Tools like Google Assistant, Apple Siri, and Amazon Alexa have created large markets for voice-enabled applications so, in theory, it should be possible to do one of three things:

  • Use default tools from the agent
  • Load existing actions for what we need
  • Build custom actions when actions are not available or they will not do what we need

For example, Google Assistant can already do voice searches in Wikipedia and Gooogle search so we have the first layer of research tools available.

We can also leverage existing voice interfaces from universities or commercial research databases.

If there are no existing actions we can develop our actions to interact with data sources and libraries. We may also want to consolidate all searches into a single command.

Search for information

Today I can tell Google Assistant to search Wikipedia for bionics and it will display the results from Wikipedia for the requested term. How difficult would it be to trigger the same search on multiple sites/engines from the same command?

Machine Learning could help us to provide a context-sensitive search interface where results are more closely aligned with the meaning of our search.

Managing Documents

Another area where we might want to use Assistant is as input For Google Docs and other applications part of GSuite. Normally I wouldn’t go with just one provider but as far as I know, Google Docs, Slides, and Sheets are the only set that is scriptable with Javascript. This would allow us to create documents programmatically from an agent so that users have to tweak and finish them.

Text to Speech

Most browsers now have text-to-speech capabilities and so do agents. It shouldn’t be too much of a stretch to use the agent’s built-in speech capabilities or piggyback on the Web Speech API to provide a more generic solution that has the browser or agent read the page to us without us having to “make” an agent to do it.

Communication

WebRTC provides both audiovisual and data channel point to point and multicasting communication.

Agents could leverage the low-level APIs or they could piggyback into existing tools like Google Meet or similar tools that allow communication and data exchange.