Epubcheck 4 is in the horizon

Epubcheck 4 is in the horizon, you should take another look at your books.

ePubcheck 4, the IDPF validator is currently in Alpha and available for download if you want test it. I downloaded it and made a first pass to what I thought was a valid book and was, not so pleasantly, surprised at how much it has changed.

I say not so pleasantly surprised because of the additonal breadth of checks the tools performs and the cryptic nature of the messages has prompted me to rethink the way in which my documents are laid out, particularly considering that I use Docbook to generate my epub content and it’s not laid out as standard HTML (uses tables to layout content wich means the table doesn’t need the same structure needed for data tables… I know… so 1990s but I did not create the stylesheets).

I’m concerned that publishers will use the new litmus test of epubcheck 4 to reject books taht would have normally passed the epubcheck 3.0.1 validation test.

The positive side of all these warnings is that documents will be more accessible because of the additional checks, assuming that authors (and publishers) bother with implementing any of the rules their books are flagged for.

For those interested, the full list of epubcheck 4 messages is listed below:

ID    Severity  Message Suggestion
ACC-001     WARNING Img and Area html elements should include an 'alt' attribute for accessibility.
ACC-002   WARNING   Input html elements should be referenced by a corresponding label element.
ACC-003   WARNING   Non-text html elements should include a 'title' attribute for accessibility.
ACC-004   WARNING   Html 'a' element must have text.
ACC-005   WARNING   Table heading cells should be identified by 'th' elements for accessibility.
ACC-006   USAGE Tables should include a 'thead' element for accessibility.
ACC-007   USAGE Epub should use epub:type attributes.
ACC-008     USAGE   Epub should have 1 landmarks nav element.
ACC-009   USAGE MathML should either have an alt text attribute or annotation-xml child element.
ACC-010   USAGE Headings should not be used within blockquote and figure elements.
ACC-011   USAGE Link elements within SVG should include an xlink:title attribute.
ACC-012   USAGE Table elements should include a caption element.
ACC-013   WARNING   Content file contains at least one inline style declaration.    Inline styles are not compatible with accessibility settings and display personalization. Use CSS Styles instead.
ACC-014   WARNING   CSS Selector font-size attribute value does not use a relative size.    Acceptable values include percentages, em multiples, 'larger', 'smaller', 'normal' or 'inherit'.
ACC-015   WARNING   CSS Selector font line-height attribute value does not use a relative size. Acceptable values include percentages, numeric multiples, em multiples, 'normal', or 'inherit'.
ACC-016   USAGE CSS Selector font-size attribute value should use a relative size.  Acceptable values include percentages, em multiples, 'larger', 'smaller', 'normal' or 'inherit'.
ACC-017   USAGE CSS Selector font line-height attribute value does not use a relative size. Acceptable values include percentages, numeric multiples, em multiples, 'normal', or 'inherit'.
CHK-001   ERROR The custom message overrides file was not found.
CHK-002   ERROR Unrecognized custom message id %1$s encountered in message overrides file '%2$s'.
CHK-003   ERROR Unrecognized custom message severity '%1$s' encountered in message overrides file '%2$s'.
CHK-004   ERROR The custom message contains too many parameters in message overrides file '%1$s'.
CHK-005   ERROR The custom suggestion contains too many parameters in message overrides file '%1$s'.
CHK-006   ERROR Unable to parse the custom format parameter in message overrides file '%1$s'.
CHK-007   ERROR Error encountered while processing custom message file '%1$s': "%2$s".
CSS-001   ERROR The '%1$s' property must not be included in an ePub Style Sheet.
CSS-002   ERROR Empty or NULL reference found.
CSS-003   ERROR Only UTF-8 and UTF-16 encodings are allowed, detected %1$s.
CSS-004   ERROR Only UTF-8 and UTF-16 encodings are allowed, detected %1$s BOM.
CSS-005   ERROR Conflicting alternate style attributes found: %1$s.
CSS-006   WARNING   CSS position:fixed property is not allowed in ePub v3 documents.
CSS-007   WARNING   Font-face reference %1$s refers to non-standard font type %2$s.
CSS-008   ERROR An error occurred while parsing the CSS: %1$s.
CSS-009   USAGE Use of certain CSS such as Columns, Transforms, Transitions, box-sizing or KeyFrames can cause pagination issues.
CSS-010   WARNING   Use of non-standard stylesheet.
CSS-011   USAGE Excessive number of css files.  Consider merging CSS files to reduce the number of CSS files.
CSS-012   USAGE Document links to multiple CSS files.
CSS-013   USAGE CSS Selector attribute is declared !Important.
CSS-015   WARNING   Alternate Stylesheet has no title.
CSS-016   WARNING   Alternate Stylesheet precedes primary stylesheet.
CSS-017   WARNING   CSS Selector specifies absolute position.
CSS-019   WARNING   CSS font-face declaration has no attributes.
CSS-020   WARNING   CSS font selector declaration uses unexpected font-size value '%1$s'.   Acceptable values include percentages, em multiples, 'larger', 'smaller', 'normal' or 'inherit'.
CSS-021   WARNING   CSS Selector font shorthand specifies an invalid System Font.   Acceptable values include 'caption', 'icon', 'menu', 'message-box', 'small-caption',  'status-bar', or 'inherit'.
CSS-022   WARNING   CSS Selector specifies global margin setting.
CSS-023   USAGE CSS Selector specifies media query.
CSS-024   USAGE CSS class Selector is not used. Remove unused CSS selectors.
CSS-025   USAGE CSS class Selector is not found.    Check for typos or define a class selector to document the use of the class.
CSS-027   USAGE CSS Selector specifies absolute position.
CSS-028   USAGE Use of Font-face declaration.
HTM-001   ERROR Any publication resource that is an XML-based media type must be a valid XML 1.0 document. XML version found: %1$s.
HTM-002   WARNING   The installed xml parser doesn't support xml version verification. Xml files must be a valid XML 1.0 document.
HTM-003   ERROR External entities are not allowed in ePub v3 documents. External entity declaration found: %1$s.
HTM-004   ERROR Irregular DOCTYPE: found '%1$s', expected '%2$s'.
HTM-005   USAGE An external reference was found.
HTM-006   USAGE XHTML Named Entities are in use.    Consider using numbered entities instead.
HTM-007   WARNING   Empty or whitespace-only value of attribute ssml:ph.
HTM-008   ERROR The src attribute is required.
HTM-009   ERROR The DOCTYPE provided is obsolete or irregular and can be removed.
HTM-011   ERROR Entity is undeclared.   Define the entity or use the numbered entity instead.
HTM-012   WARNING   Found a link to a CFI in an external book.
HTM-013   USAGE Intra-Publication CFIs found in document.
HTM-014   WARNING   Invalid file extension for HTML5 file, expecting (html, htm or xhtml).
HTM-015   WARNING   HTML4 DOCTYPE definition within ePub v3.
HTM-016   WARNING   HTML5 DOCTYPE definition within ePub v2.
HTM-017   ERROR Content file has different language value in attributes xml:lang and lang.
HTM-018   ERROR Content file has invalid language value at attribute xml:lang.
HTM-019   ERROR Content file has invalid language definition at attribute lang.
HTM-020   USAGE Content file doesn't contain xml:lang attribute.
HTM-021   USAGE Content file doesn't contain lang attribute.
HTM-022   USAGE Document contains unnecessary DIV or SPAN tags. Merge DIV or SPAN tags when they are consecutive and use the same style.
HTM-025   WARNING   Non-registered URI scheme type found in href.
HTM-027   WARNING   Lists should include 2 or more list items.  Lists should have more than one item for accessibility.
HTM-028   WARNING   Input html elements should include an id.
HTM-029   WARNING   Label html elements should include a 'for' attribute referencing the id of an input element.
HTM-033   WARNING   Html head element does not have a title.
HTM-036   WARNING   IFrames are highly discouraged.
HTM-038   USAGE Ensure b, i, em, and strong elements are used in compliance with W3C HTML5 directives.  CSS styles are usually more appropriate for italics or bold text.
HTM-043   USAGE SVG elements should include an xml:lang and lang attributes.
HTM-044   USAGE Namespace uri '%1$s' was included but not used.
HTM-045   INFO  Encountered empty href.
HTM-046   ERROR Fixed format item has no viewport defined.
HTM-047   ERROR Html viewport is missing height and/or width.
HTM-048   ERROR SVG ViewBox is missing on fixed format document.
MED-001   ERROR Video poster must have core media image type.
MED-002   ERROR %1$s element doesn't provide fallback.
MED-003   ERROR Non-standard image resource of type %1$s found.
MED-004   ERROR Image file header may be corrupted.
MED-005   ERROR Media Overlay audio reference %1$s to non-standard audio type %2$s found.
MED_006   WARNING   Some browsers do not support rendering SVG images which use a filename in the xlink:href property.
NAV-001   ERROR The nav file is not supported for ePub v2.
NCX-002   ERROR  element is not defined.
NCX-003   USAGE An .NCX file is required for TOC navigation on ePub v2 readers.
OPF-001   ERROR There was an error when parsing the ePub version: %1$s.
OPF-002   FATAL The OPF file '%1$s' was not found in the ePub.
OPF-003   WARNING   Item '%1$s' exists in the ePub, but is not declared in the OPF manifest.
OPF-004   ERROR Invalid prefix '%1$s'.
OPF-005   ERROR URL for prefix '%1$s' doesn't exist.
OPF-006   ERROR URL expected for prefix '%1$s'. Check the formatting of the prefix attribute of the package tag and ensure there is no extraneous whitespace.
OPF-007   ERROR Redeclaration of reserved prefix '%1$s'.
OPF-008   ERROR Handler binding for core Media-type '%1$s' is not allowed.
OPF-009   ERROR The media-type %1$s has already been assigned handler '%2$s'.
OPF-010   ERROR Error resolving reference: '%1$s'.
OPF-011   ERROR itemref can't have both page-spread-right and page-spread-left properties.
OPF-012   ERROR Item property '%1$s' is not defined for media type '%2$s'.
OPF-013   ERROR The type property '%1$s' on the object tag does not match the declared media-type '%2$s' in the OPF manifest.
OPF-014   ERROR The property '%1$s' should be declared in the OPF file.
OPF-015   ERROR The property '%1$s' should not be declared in the OPF file.
OPF-016   ERROR The element "rootfile" is missing its required attribute "full-path".
OPF-017   ERROR The attribute "full-path" on element "rootfile" must not be empty.
OPF-018   WARNING   The 'remote-resources' property was declared in the OPF, but no reference to remote resources has been found. Make sure this property is legitimate.
OPF-019   FATAL Spine tag was not found in the OPF file.
OPF-020   WARNING   Excessive number of spine items.
OPF-021   WARNING   Use of non-registered URI scheme type in href: '%1$s'.
OPF-022   ERROR Invalid path: '%1$s'.
OPF-023   ERROR Expected 1 metadata definition tag but found '%1$s'.
OPF-024   ERROR Found unknown ePub version %1$s.
OPF-025   ERROR Property '%1$s' can take only one value.
OPF-026   ERROR Found malformed property value: '%1$s'.
OPF-027   ERROR Undefined property: '%1$s'.
OPF-028   ERROR Undeclared prefix: '%1$s'.
OPF-029   ERROR The file '%1$s' does not appear to match the media type %2$s, as specified in the OPF file.
OPF-030   ERROR The unique-identifier '%1$s' was not found.
OPF-031   ERROR File listed in reference element in guide was not declared in OPF manifest: %1$s.
OPF-032   ERROR Guide references '%1$s' which is not a valid 'OPS Content Document'.
OPF-033   ERROR The spine contains no linear resources.
OPF-034   ERROR The spine contains multiple references to the manifest item with id '%1$s'.
OPF-035   WARNING   Media type 'text/html' is not appropriate for XHTML/OPS.    Use 'application/xhtml+xml' instead.
OPF-036   WARNING   Media type '%1$s' is not an appropriate video mimetype. Use 'video/mp4' or 'video/h264' instead.
OPF-037   WARNING   Found deprecated media-type '%1$s'.
OPF-038   WARNING   Media type '%1$s' is not appropriate for an OEBPS 1.2 context; Use 'text/x-oeb1-document' instead.
OPF-039   WARNING   Media-type '%1$s' is not appropriate in an OEBPS 1.2 context. Use 'text/x-oeb1-css' instead.
OPF-040   ERROR Fallback item could not be found.
OPF-041   ERROR Fallback-style item could not be found.
OPF-042   ERROR '%1$s' is not a permissible spine media-type.
OPF-043   ERROR Spine item with non-standard media-type '%1$s' has no fallback.
OPF-044   ERROR Spine item with non-standard media-type '%1$s' has a fallback to non-standard media-type.
OPF-045   ERROR Encountered circular reference in fallback chain.
OPF-046   ERROR Scripted property is not set on mediaType handler.
OPF-047   USAGE OPF file is using OEBPS 1.2 syntax allowing backwards compatibility.
OPF-048   ERROR Package tag is missing its required unique-identifier attribute and value.
OPF-049   ERROR Item id '%1$s' was not found in the manifest.
OPF-050   ERROR TOC attribute references resource with non-NCX mime type; 'application/x-dtbncx+xml' is expected.
OPF-051   WARNING   Image dimensions exceed recommended size.
OPF-052   ERROR Role value '%1$s' is not valid.
OPF-053   WARNING   Date value '%1$s' does not follow recommended syntax as per http://www.w3.org/TR/NOTE-datetime:%2$s.
OPF-054   ERROR Date value '%1$s' is not valid as per http://www.w3.org/TR/NOTE-datetime:%2$s.
OPF-055   WARNING   %1$s tag is empty.
OPF-056   WARNING   Media type '%1$s' is not an appropriate audio mimetype. Use 'audio/mp3', 'audio/mp4' or 'audio/ogg' instead.
OPF-057   WARNING   Image file length exceeds recommended size.
OPF-058   USAGE Spine item has no TOC entry reference.  Every spine item in the manifest should be referenced by at least one TOC entry.
OPF-059   USAGE Spine item has no NAV entry reference.  Every spine item in the manifest should be referenced by at least one NAV entry.
OPF-060   ERROR Duplicate entry in the ZIP file: '%1$f'.
OPF-061   WARNING   Duplicate entry in the ZIP file (after Unicode NFC normalization) '%1$f'.
PKG-001   WARNING   Validating the ePub against version %1$s but detected version %2$s.
PKG-003   ERROR Unable to read ePub file header.  This is likely a corrupted ePub file.
PKG-004   FATAL Corrupted ePub ZIP header.
PKG-005   ERROR The mimetype file has an extension of length %1$s.  No filename extensions are permitted for the mimetype file.
PKG-006   ERROR Mimetype file entry is missing or is not the first file in the archive.
PKG-007   ERROR Mimetype file should only contain the string 'application/epub+zip'.
PKG-008   FATAL Unable to read file '%1$s'.
PKG-009   ERROR File name contains characters that are not allowed in OCF file names: '%1$s'.
PKG-010   WARNING   Filename contains spaces, therefore URI escaping is necessary. Consider removing spaces from filename.
PKG-011   ERROR Filename is not allowed to end with '.'.
PKG-012   WARNING   File name contains the following non-ascii characters: %1$s. Consider changing the filename.
PKG-013   ERROR The ePub file includes multiple OPS renditions.
PKG-014   WARNING   The ePub contains empty directory '%1$s'.
PKG-015   FATAL Unable to read ePub contents: %1$s
PKG-016   WARNING   Use only lowercase characters for the ePub file extension for maximum compatibility.    For maximum compatibility, use '.epub'.
PKG-017   WARNING   Uncommon ePub file extension.   For maximum compatibility, use '.epub'.
PKG-018   FATAL The ePub file is not found.
PKG-020   ERROR OPF file '%1$s' is not found.
PKG-021   ERROR Corrupted image file encountered.
RSC-001   ERROR File '%1$s' is not found.
RSC-002   FATAL Required META-INF/container.xml resource is not found.
RSC-003   ERROR No rootfile tag with media type 'application/oebps-package+xml' was found in the container.
RSC-004   ERROR File '%1$s' could not be decrypted.
RSC-005   ERROR Error while parsing file '%1$s'.
RSC-006   ERROR Remote resource reference not allowed; resource must be placed in the OCF.  Only audio and video remote resources are permitted.
RSC-007   ERROR Referenced resource is not found in the ePub.
RSC-008   ERROR Referenced resource is not declared in the OPF manifest.
RSC-009   ERROR A fragment identifier should not be used with an img src attribute.
RSC-010   ERROR Reference to non-standard resource type found.
RSC-011   WARNING   Found a reference to a resource that is not a spine item.
RSC-012   ERROR Fragment identifier is not defined.
RSC-013   ERROR Fragment identifier is used in a reference to a stylesheet resource.
RSC-014   ERROR Fragment identifier defines an incompatible resource type.
RSC-015   ERROR A fragment identifier is required for svg use tag references.
RSC-016   FATAL Fatal Error while parsing file '%1$s'.
RSC-017   WARNING   Warning while parsing file '%1$s'.
RSC-018   WARNING   Altimg file '%1$s' is not found.
SCP-001   USAGE Use of Javascript eval() function in ePub scripts is a security risk.
SCP-002   USAGE Use of XMLHttpRequest in ePub scripts is a security risk.
SCP-003   USAGE Local and Session Storage is not currently supported.
SCP-004   ERROR Content file contains script which is not supported in ePub v2.
SCP-005   ERROR Content file contains script but it is not marked as scripted.
SCP-006   USAGE Inline scripts found.
SCP-007   WARNING   Script references 'innerHtml'.  Use a DOM instead.
SCP-008   WARNING   Script references 'innerText'.  Use 'textContent' instead.
SCP-009   USAGE Content file uses mouse event handlers. Ensure that all mouse driven functionality is accessible from the keyboard and touch devices.

Visualizing CSS properties

One of my earliest experiments in data visualization was to create visualization of the CSS properties as they are documented in the Web Platform Documentation project. Now that it’s in a shape where I’m willing to let it out in the wild it’s time to write about it and explain the rationale and the technology.

Rationale

I’m a visual person. Rather than search for something that I may or may not know exactly what it is, I’d rather look at something that, I hope, will make it easier for me to find what I’m looking for.

I’m also lazy. Instead of looking for a property in one place and then manually typing the full URL of the Web Platform Documentation project I’d rather add the URLs for all properties directly to the visualization so that, when the I find the property he’s looking for, I can go directly to it from the visualization tree.

Building the visualization

The data

The first thing I did was to pull the data from the Web Platform Documentation project using their API to generate an initial JSON file. I then had to edit the file manually to produce something closer to the JSON format that I was looking for:

{
    "name": "CSS",
    "children": [
        {
            "name": "Alignment",
            "children": [
                {
                    "size": 1000,
                    "name": "align-content",
                    "url": "http://docs.webplatform.org/wiki/css/properties/align-content"
                },
                {
                    "size": 1000,
                    "name": "align-items",
                    "url": "http://docs.webplatform.org/wiki/css/properties/align-items"
                },
                {
                    "size": 1000,
                    "name": "align-self",
                    "url": "http://docs.webplatform.org/wiki/css/properties/align-self"
                },
                {
                    "size": 1000,
                    "name": "alignment-adjust",
                    "url": "http://docs.webplatform.org/wiki/css/properties/alignment-adjust"
                },
                {
                    "size": 1000,
                    "name": "alignment-baseline",
                    "url": "http://docs.webplatform.org/wiki/css/properties/alignment-baseline"
                }
            ]
        }
    ]
}

Every time I edited or made a change to the JSON file (the resulting full JSON file is about 2500 lines of code) I ran it through JSON lint to make sure that the resulting content was valid JSON. I haven’t always done this and it has been a constant source of problems: The page appears blank, only part of the content is displayed and other anoyances that took forever to correct.

Once we have the JSON file working, we can move into the D3 code.

The technology

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.
From http://d3js.org

What this means is that we can build visual content based on data have collected or arbitrary data we have available. In this case we are visualizing an arbitrary grouping of CSS properties from the Web Platform Documentation project; all properties are listed but the grouping may change as the group comes to a consensus regarding the groups.

D3 follows a fairly straightforward process. We start by defining all our variables at the top of the script to prevent Javascrp variable hoisting. The code looks like this:

// Starting values were:
// width: 2140 - margin.right - margin.left
// height : 1640 - margin.top - margin.bottom
var margin = {top: 20, right: 120, bottom: 20, left: 120},
    width = 1070 - margin.right - margin.left,
    height = 820 - margin.top - margin.bottom;

var i = 0,
    duration = 750,
    root

var tree = d3.layout.tree()
    .size([height, width]);

var diagonal = d3.svg.diagonal()
    .projection(function(d) { return [d.y, d.x]; });

We create the SVG-related elements that we need in order to display the visualization data. The steps in the code below are:

  • Select the body of the document
  • Append the svg element
  • Set up width and heigh attributes with default values
  • Create a SVG group (indicated by the <g> tag) and translate it (move it by the ammount indicated by the top and left margin)
var svg = d3.select("body")
  .append("svg")
    .attr("width", width + margin.right + margin.left)
    .attr("height", height + margin.top + margin.bottom)
  .append("g")
    .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

We load the JSON file using D3′s JSON loader and set up the root element and its position.

The function collapse makes sure tha elements without children are collapsed when we first open the visualization page. I wanted to make sure that users would not be overwhelmed with all the information available in the visualization and had a choice as to what items they would click and what information they’d access.

Preventing children from automatically displaying also prevents the clutter of the tree. If there are too many children open the vertical space gets reduced and it becomes hard to distinguish which item we are clicking in.

I’ve also set a default height for all elements… 100px sounds good at this stage.

d3.json("json/css-data.json", function(error, css) {
  root = css;
  root.x0 = height / 2;
  root.y0 = 0;

function collapse(d) {
    if (d.children) {
      d._children = d.children;
      d._children.forEach(collapse);
      d.children = null;
    }
  }  root.children.forEach(collapse);
  update(root);
});

d3.select(self.frameElement).style("height", "100px");

Because the click will change the nature of the layout and the number of visible elements we need to update the layout everytime the user clicks on a valid element. This wil involve hidding old elements,showing showing new nodes in the tree.

function update(source) {

// Compute the new tree layout.
  var nodes = tree.nodes(root).reverse(),
      links = tree.links(nodes);

// Normalize for fixed-depth.
  nodes.forEach(function(d) { d.y = d.depth * 200; });

// Update the nodes…
  var node = svg.selectAll("g.node")
      .data(nodes, function(d) { return d.id || (d.id = ++i); });

// Enter any new nodes at the parent's previous position.
  var nodeEnter = node.enter().append("g")
      .attr("class", "node")
      .attr("class", function(d) {
        if (d.children) {
          return "inner node"
        } else {
          return "leaf node"
        }
      })
      .attr("transform", function(d) { return "translate(" + source.y0 + "," + source.x0 + ")"; })
      .on("click", click);

Using D3′s Enter/Append/Exist system we go back into the nodes we created, we append a new circle and set its radius and color (lines 1 – 3 in the code below).

Next I add the text for each node, set up their X (line 5) and Y (line 6) coordinates for the text node. I’ve aligned the text usin a D3 trick where setting the Y value to .35em centers the text vertically.

For each leaf node I set up a link as only elements without children have URL attributes. We do this in two steps:

  • Append a SVG anchor element (svg:a) which is different than our regular HTML anchor (line 7)
  • Add an Xlink, the XML vocabulary for defining links between resources (line 8) using the xlink:href syntax

Finally, we setup and place the linkend attribute for each node in such a way that nodes with children will display their text to the left of the assigned circle and nodes without children will display the text to the right of the circle (line 11)

  nodeEnter.append("circle")
    .attr("r", 1e-6)
    .style("fill", function(d) { return d._children ? "lightsteelblue" : "#fff"; });
    nodeEnter.append("text")
      .attr("x", function(d) { return d.children || d._children ? -10 : 10; })
      .attr("dy", ".35em")
      .append("svg:a")
      .attr("xlink:href", function(d){return d.url;})
      .style("fill-opacity", 1)
      .text(function(d) { return d.name; })
    .attr("text-anchor", function(d) { return d.children || d._children ? "end" : "start"; });

Most of the remaining work is to transition elements to and from their current position. This would be so much easier if we were using a library such as jQuery or Dojo but the result is worth the additional code.

The duration for all transitions is hardcoded to 750 miliseconds. Whether duration affects the user experiecne is an area to look further into.

// Transition nodes to their new position.
  var nodeUpdate = node.transition()
      .duration(duration)
      .attr("transform", function(d) { return "translate(" + d.y + "," + d.x + ")"; });

nodeUpdate.select("circle")
      .attr("r", 4.5)
      .style("fill", function(d) { return d._children ? "lightsteelblue" : "#fff"; });

nodeUpdate.select("text")
      .style("fill-opacity", 1);

// Transition exiting nodes to the parent's new position.
  var nodeExit = node.exit().transition()
      .duration(duration)
      .attr("transform", function(d) { return "translate(" + source.y + "," + source.x + ")"; })
      .remove();

nodeExit.select("circle")
      .attr("r", 1e-6);

nodeExit.select("text")
      .style("fill-opacity", 1e-6);

// Update the links…
  var link = svg.selectAll("path.link")
      .data(links, function(d) { return d.target.id; });

// Enter any new links at the parent's previous position.
  link.enter().insert("path", "g")
      .attr("class", "link")
      .attr("d", function(d) {
        var o = {x: source.x0, y: source.y0};
        return diagonal({source: o, target: o});
      });

// Transition links to their new position.
  link.transition()
      .duration(duration)
      .attr("d", diagonal);

// Transition exiting nodes to the parent's new position.
  link.exit().transition()
      .duration(duration)
      .attr("d", function(d) {
        var o = {x: source.x, y: source.y};
        return diagonal({source: o, target: o});
      })
      .remove();

// Stash the old positions for transition.
  nodes.forEach(function(d) {
    d.x0 = d.x;
    d.y0 = d.y;
  });

The final bit of magic is to use D3′s onClick event to toggle the display of our content.

var svg
// Toggle children on click.
function click(d) {
  if (d.children) {
    d._children = d.children;
    d.children = null;
  } else {
    d.children = d._children;
    d._children = null;
  }
  update(d);
}

Where to go next?

There are some areas I want to further explore as I move forward with the visualization and learn more about how to visualize data:

  • Does the length of the transitions change the way people react to the data?
  • How can we control the space between items when they are too many open?

I will post the answers to these questions as I find the answers :-)

HTML as a single source format

In this essay I will take what may be an unpopular position: Using HTML with XML syntax (XHTML) is currently the best format to put your content in because it is easier to convert from XHTML/CSS to pretty much any other format. In making this case we’ll explore and work in the following areas and answer the following questions:

Definitions

When we speak about XHTML in this document we refer to an HTML document using XHTML syntax. I will not change the mime type on the server to fully comply with XHTML restrictions.

Why XHTML

The two main reasons I advocate XHTML as an authoring format are

XHTML enforces code clarity and authoring discipline

XHTML limits the freeform structure of HTML5. Documents conforming to XHTML specifications must have, at a minimum:

  • A DOCTYPE declaration
  • An HTML element
  • A HEAD element
  • TITLE element
  • BODY element

The structure written as XHTML tags looks like this:

<!DOCTYPE html>
  <html>
    <head>
      <title>Title Goes Here<title>
    <head>
    <body>
      <h1>Content Area</h1>
    <body>
  <html>

This minimal structure must comply with the requirements below

All XHTML tag names & attribute names must be in lowercase

All XHTML attributes and elements must be in lower case

The following elements are not legal XHTML:

<DIV CLASS="chapter">Chapter 1</div>

<Div Class="chapter">Chapter 1</div>

All XHTML elements must close

All elements must be closed, this includes both our standard tags such as the paragraph tag

<p>This is a paragraph</p>

to empty elements such as images and form inputs elements

<img src="images/test.png" height="800" width="600" alt="Test image" />

<input type="submit" value="Submit" />

All XHTML elements must be properly nested

XHTML insists on proper nesting of the elements on our content. This is no longer legal

<p>This is the content of a paragraph

<p>This is our second paragraph

And it should be writen like this:

<p>This is the content of a paragraph</p>

<p>This is our second paragraph</p>

All XHTML attribute values must be quoted

In addition to being lowercased, attributes must be quoted. Rather than:

<div class=chapter>Chapter 1</div>

It has to be written like this:

<div class="chapter">Chapter 1</div>

Because it is structured, we can use transformation tools to convert to/from XHTML

A lot of the discussions I’ve had with people seem to focus in the drawbacks of XHTML format as end users. One of the strengths W3C cited when moving to XHTML as the default format for the web was how easy it was for machines to read it and covert it to other formats.

I’ll cover two examples using Markdown: straigth transformation and converting Markdown into templated XHTML and an example of using XSLT 1.0 to covert one flavor of XHTML into another using Xsltproc

From markdown to html, straight up

One of the goals of Markdown is to: allow you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML). The original tool and all its ttranslations to other languages are built to allow the conversion; where they are different is in the number of extensions to the core markdown language and the language the tools themselves are written on.

For these examples I chose Python Markdown mostly because it’s the language and the tool I’m familiar with. We will use the markdown file for the Markdown home page at Daring Fireball http://daringfireball.net/projects/markdown/index.text

Below is a portion of the resulting XHTML code:

<h2>Introduction</h2>

<p>Markdown is a text-to-HTML conversion tool for web writers. Markdown
allows you to write using an easy-to-read, easy-to-write plain text
format, then convert it to structurally valid XHTML (or HTML).</p>

<p>Thus, "Markdown" is two things: (1) a plain text formatting syntax;
and (2) a software tool, written in Perl, that converts the plain text
formatting to HTML. See the <a href="/projects/markdown/syntax">Syntax</a> page for details pertaining to
Markdown's formatting syntax. You can try it out, right now, using the
online <a href="/projects/markdown/dingus">Dingus</a>.</p>

<p>The overriding design goal for Markdown's formatting syntax is to make
it as readable as possible. The idea is that a Markdown-formatted
document should be publishable as-is, as plain text, without looking
like it's been marked up with tags or formatting instructions. While
Markdown's syntax has been influenced by several existing text-to-HTML
filters, the single biggest source of inspiration for Markdown's
syntax is the format of plain text email.</p>

<p>The best way to get a feel for Markdown's formatting syntax is simply
to look at a Markdown-formatted document. For example, you can view
the Markdown source for the article text on this page here:
<a href="http://daringfireball.net/projects/markdown/index.text">http://daringfireball.net/projects/markdown/index.text</a></p>

The conversion process itself is simple. Using the Perl version it looks like this:

markdown content/webgl-2d-scale.md > test.html

From Markdown to templated XHTML

As part of my sunshine-markdown project I’ve researched ways to convert markdown to XHTML. as

[10:30:54] carlos@rivendell sunshine-markdown 4826$ ./sunshine --verbose
processing:  content/webgl-2-textures.md
processing:  content/webgl-2d-matrices.md
processing:  content/webgl-2d-rotation.md
processing:  content/webgl-2d-scale.md
processing:  content/webgl-2d-translation.md
processing:  content/webgl-2d-vs-3d-library.md
processing:  content/webgl-3d-camera.md
processing:  content/webgl-3d-orthographic.md
processing:  content/webgl-3d-perspective.md
processing:  content/webgl-3d-textures.md
processing:  content/webgl-and-alpha.md
processing:  content/webgl-animation.md
processing:  content/webgl-boilerplate.md
processing:  content/webgl-fundamentals.md
processing:  content/webgl-how-it-works.md
processing:  content/webgl-image-processing-continued.md
processing:  content/webgl-image-processing.md
processing:  index.md 

Sunshine is hardcoded to put the content of each markdown file into a template that looks something like this:

&lt; ?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"
  xmlns:epub="http://www.idpf.org/2007/ops"
  xmlns:m="http://www.w3.org/1998/Math/MathML"
  xmlns:pls="http://www.w3.org/2005/01/pronunciation-lexicon"
  xmlns:ssml="http://www.w3.org/2001/10/synthesis"
  xmlns:svg="http://www.w3.org/2000/svg">
<!-- this file is auto-generated from %(src_file_name)s. Do not edited directly -->
<!--
Copyright 2014, Carlos Araya
See: http://caraya.mit-license.org/ for details on the license and copyright release
-->
<head>
  <meta charset="utf-8"/>
  <title>%(title)s</title>
</head>
<body>

<section epub:type="chapter">
    <h1>%(title)s</h1>
    %(content)s
  </section>

</body>
</html>

Using XSLT to convert XHTML into ePub-ready XHTML

One of the things we forget is that, because XHTML is structured content we can use XSLT and XPATH to convert it to other XML-based dialects, such as the XHTML dialect required for ePub3 conformance. A basic template to convert a div into a section with the proper attributes for ePub work may look something like this:

&lt; ?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <!-- Since we are expecting one chapter per file we can match the root element to begin -->
  <xsl:template match="/">
    <div class="featured">
      <xsl:apply -templates match="div[@epub:type='chapter']"></xsl:apply>
    </div>
  </xsl:template>
  <!-- The second template matches div with epub:type attribute of chapter -->
  <xsl:template match="div[@epub:type='chapter']">
    <xsl:apply -templates></xsl:apply>
  </xsl:template>
</xsl:stylesheet>

Flex Boxes and the Holy Grail

Content and images taken from Mozilla Developer Network Flexbox Page and A Complete Guide to Flexbox

One of the hardest things to do in web design is to create a fluid 3 column layout, also known as the holy grail. I could talk for hours about how it was goo enough for the technologies available when the model was first developed, how much of a pain it is to implement correctly, how it did not support mobile and small form devices and how long have developers wanted a solution.

Example of Holy Grail Layout
Example of Holy Grail Layout. Courtesy of Litesite

Instead I will talk about a solution that has finally become widely available. With the release of Firefox 28 to wide availability, the last of the big 4 browsers now supportd the full flexible box layouts specificactions. This means we can do some awesome crazy things with layouts that, until now we could only dream about.

In this article I will explain what Flexboxes are, the terminology, syntax, new CSS required to make it work and end with reworking the Holy Grail Layout using Flexboxes.

Getting things setup

Flexboxes don’t use the same terminology as other CSS elements. We need to discuss this terminology before we can fully understand and use the new layout in our own work.

Flex Container

The outermost container for our Flex Layout. We create this by assigning flex or inline-flex as the values of the display property depending on whether you want to use the Flex Container as a block or inline element.

.flex-container {
  display: flex
  /*  If you want the container to be a block element*/
}

or

.container {
  display: inline-flex
  /*  If you want the container to be an inline element*/
}

Each Flex Box container defines two axes:

  • A main axis that defines how the elements flow in the container
  • A cross axis that is perpendicular to the main

flex-direction

The flex-direction attribute defines the main axis (start, direction, end) for the container. It can take one of 4 values:

  • row: The flex container’s main-axis is defined to be the same as the text direction. The elements in the container flow in the same direction as the page content (left to right in the case of English)
  • row-reverse: The flex container’s main-axis is defined to be the same as the text direction. The elements in the container flow in the opposite direction to the page content (right to left in the case of English)
  • column: The flex container’s main-axis is defined to be the same as the block-axis. The main-start and main-end points are the same as the before and after points of the writing-mode (top to bottom in English)
  • column-reverse: The flex container’s main-axis is defined to be the same as the block-axis. The main-start and main-end points are reversed from the before and after points of the writing-mode (bottom to top in English)
.content {
  flex-direction: row;
}

flex-wrap

  • nowrap (default): single-line / left to right in ltr; right to left in rtl
  • wrap: multi-line / left to right in ltr; right to left in rtl
  • wrap-reverse: multi-line / right to left in ltr; left to right in rtl
.content {
  flex-direction: row;
  flex-wrap: wrap;
}

flex-flow

This is a shortcut for flex-direction and flex-wrap which together define both the main and cross axes. The default value is:

.container {
  flex-flow: "row nowrap"
}

justify-content

Justify content takes a special meaning when used in flex boxes: It tells the browser how to arrange the cintent of the container in its main axis. It can take the following values:

  • flex-start (default): items are packed toward the start line
  • flex-end: items are packed toward to end line
  • center: items are centered along the line
  • space-between: items are evenly distributed in the line; first item is on the start line, last item on the end line
  • space-around: items are evenly distributed in the line with equal space around them
.content {
  flex-direction: row;
  justify-content: space-around;
}

align-items

align-items controls the vertical positioning of the row’s element. It is the vertical counterpart to align-items. It can take the following values:

  • flex-start: cross-start margin edge of the items is placed on the cross-start line
  • flex-end: cross-end margin edge of the items is placed on the cross-end line
  • center: items are centered in the cross-axis
  • baseline: items are aligned such as their baselines align
  • stretch (default): stretch to fill the container (still respect min-width/max-width)

align-content

Controls the use of the rermaining vertical space within a flex-container’s row. This has no effect if there is only one row on the

  • flex-start: lines packed to the start of the container
  • flex-end: lines packed to the end of the container
  • center: lines packed to the center of the container
  • space-between: lines evenly distributed; the first line is at the start of the container while the last one is at the end
  • space-around: lines evenly distributed with equal space between them
  • stretch (default): lines stretch to take up the remaining space

Flex Item

Every element inside our Flex Container becomes a Flex Item. When we put text on the page as a direct child of the Flex Box it’ll be wrapped on an implicit flex item. There is no need to declare these elements explicitly.

order

Indicates the order (positive numbers without a unit) in which the items in a row are displayed. The default is to display them in source order.

.content-item1{
  order: 1;
}

flex-grow

Indicates the proportion that an item can grow by if necessary.

If all items have flex-grow set to 1, every child will set to an equal size inside the container. If you were to give one of the children a value of 2, that child would take up twice as much space as the others.

In the example below, item2 will take twice as much space as item1 and item3 if more space becomes available:

#item1 {
  grow: 1;
}
#item2{
  grow: 2;
}

#item3{
  grow: 1;
}

flex-shrink

Indicates the proportion that an item can shrink by if necessary.

If all items have flex-shrink set to 1, every child will set to an equal size inside the container. If you were to give one of the children a value of 2, that child would take up twice as much space as the others.

flex-basis

This defines the default size of an element before the remaining space is distributed.

Supported values:

  • length
  • auto

The default is auto

flex

This is the shorthand for flex-grow, flex-shrink and flex-basis. The flex-shrink and flex-basis parameters are optional.

Default is 0 1 auto.

align-self

If this property is set, it will override the align-item property of the parent.

  • flex-start: cross-start margin edge of the items is placed on the cross-start line
  • flex-end: cross-end margin edge of the items is placed on the cross-end line
  • center: items are centered in the cross-axis
  • baseline: items are aligned such as their baselines align
  • stretch (default): stretch to fill the container (still respect min-width/max-width)

The Holy Grail with Flex Boxes

Working example in Codepen:

See the Pen First Attempt at Flexbox by Carlos Araya (@caraya) on CodePen.

Browser compatibility

Taken from Mozilla Developer Network Flexbox Page

Desktop Browsers

Feature Firefox (Gecko) Chrome Internet Explorer Opera Safari
Basic support (single-line flexbox) 18.0 (18.0)-moz(Behind a pref) [2]
22.0 (22.0) [2]
21.0-webkit
29.0
11 [3] 12.10
15-19 -webkit
6.1-webkit [1]
Multi-line flexbox 28.0 (28.0) 21.0-webkit
29.0
11 [3] 12.10
15-19 -webkit
6.1-webkit [1]

Mobile Browsers

Feature Firefox Mobile (Gecko) Android IE Phone Opera Mobile Safari Mobile
Basic support (single-line flexbox) 18.0 (18.0)-moz(Behind a pref) [2]
22.0 (22.0) [2]
? ? 15-19 -webkit 7-webkit [1]
Multi-line flexbox 28.0 (28.0) ? ? 15-19 -webkit 7-webkit [1]

1

Notes

[1] Safari up to 6.0 ( 6.1 for iOS ) supported an old incompatible draft version of the specification. Safari 6.1( 7 for iOS ) has been updated to support the final version

[2] Up to Firefox 22, to activate flexbox support, the user has to change the about:config preference “layout.css.flexbox.enabled” to true. From Firefox 22 to Firefox 27, the preference is true by default, but the preference has been removed in Firefox 28

[3] Internet Explorer 10 supports an old incompatible draft version of the specification; Internet Explorer 11 has been updated to support the final version.

Getting better at Javascript

I first came accross this deck while browsing Rebecca’s blog a few months ago. It made me think a lot about how can I write more Javascript and how can I build projects around my code writing.

Being at Fluent has made me think a lot about the projects I’m working on and how best to continue leveraging my coding with those things I’m interested in.

I’ll copy the expanded quote from Paul Graham that Rebecca uses on her presentation:

It takes confidence to throw work away. You have to be able to think, there’s more where that came from. When people first start drawing, for example, they’re often reluctant to redo parts that aren’t right; they feel they’ve been lucky to get that far, and if they try to redo something, it will turn out worse. Instead they convince themselves that the drawing is not that bad, really– in fact, maybe they meant it to look that way.

Dangerous territory, that; if anything you should cultivate dissatisfaction. In Leonardo’s drawings there are often five or six attempts to get a line right. The distinctive back of the Porsche 911 only appeared in the redesign of an awkward prototype. In Wright’s early plans for the Guggenheim, the right half was a ziggurat; he inverted it to get the present shape.

Mistakes are natural. Instead of treating them as disasters, make them easy to acknowledge and easy to fix. Leonardo more or less invented the sketch, as a way to make drawing bear a greater weight of exploration. Open-source software has fewer bugs because it admits the possibility of bugs.

It helps to have a medium that makes change easy. When oil paint replaced tempera in the fifteenth century, it helped painters to deal with difficult subjects like the human figure because, unlike tempera, oil can be blended and overpainted.

Creating the right kind of wow

I had to see this presentation mulitple times yesterday and today before I realized what she actually meant. I’ve been guilty of wanting the WOW! the immediate surprise at the latest and greates bells and whistles.

But that’s not what she meant and that’s not what we should be concentrating on. We should be concentrating on the wow that makes people change their view of a subject or topic.