TWiT.tv 4.0

Why A New Site

The Internet is changing. And so is TWiT.tv. The way websites look seems to change every few months, but even the way they work, and the very reason they exist, has shifted since the first pages at TWiT.tv went up in 2005.

I designed and implemented that original site by myself, choosing Drupal 4 as the content management system. It didn't take long, as we added more shows and more hosts, for TWiT to outgrow that very simple original design. I asked Amber MacArthur, host of Inside The Net on TWiT, and her brother Jeff, to redesign the site. Their company, MGImedia Communications, now Konnekt, crafted a new look and user interface for us and I hired the best Drupal programmers I knew, Lullabot, to implement it. TWiT.tv v2.0, engineered chiefly by Ted Serbinski, went live in 2007 and served us well for four years. But changes in our business and intense growth prompted another redesign by ImageX in 2011. By the end of 2014 I had decided it was time for another redesign, but this time I wanted to undertake something more ambitious.

In talks about new media for more than 10 years I've been saying that our job was to provide content for our audience when they wanted, where they wanted, in any form they wanted, without limitations on how they could consume it (I remember telling that to a convention of HBO employees in 2004 and almost being shown the door. But even HBO is coming around.) One of the reasons TWiT is ad-supported free media is because it let us do that. It was in our best interest to put our shows everywhere, to serve them up live or on-demand, in audio or video, and everywhere you wanted to watch without copy protection or paywalls.

In the early days of TWiT that meant we needed a website to provide you with a directory of shows and RSS feeds. Most people, once they found out about TWiT, went to iTunes to subscribe. And that was how 90% of our audience listened for years.

Today the market is much more fragmented. More than half of the visitors to our site are using phones or tablets. iTunes downloads are now only about 16% of the total. Another 10% of our audience watches the live stream either on the website or by plugging the URL into a video player like VLC. 30% download pre-recorded shows via the website. All the rest, nearly half, use apps to watch or listen on their mobile phones, tablets, Rokus, TiVos, Samsung TVs, many other platforms. That's good. We've put TWiT in front of our audiences wherever they want it, but it means that our website plays a very different role as apps and mobile devices become increasingly important.

As I began to appreciate this drastic change in how you watch and listen, I came to the realization that we needed to pay more attention to mobile, both on the website and via apps. I, and many others, began to think of what we do as not broadcasting, or podcasting, but content as a service, or CaaS. That's a pretty high-falutin' term, and so new there's no Wikipedia entry for it, but it does describe what we're trying to do.

Content as a Service

RSS feeds were a step in the right direction, making it possible for readers and listeners to subscribe to a feed and get content in any manner they wanted. Instead of forcing you to read the news in a newspaper or on a web page, RSS feeds let you decide where you'd read and in what form. But RSS doesn't go far enough. We are taking it a step farther and creating a full-blown API that can deliver our content, all the meta-data associated with it, and additional information about our live schedule, hosts, sponsors, and events on-demand, to any app that wants it.

Our new website is just one consumer of this API. We're developing new apps on iOS, Android, and Windows that will also use the API. And we're inviting anyone who knows how to make a JSON call to create their own sites and apps. What we're unveiling today is much more than a new site, it's a new way to deliver content that fulfills my goal to give our audience the shows they want, when they want, where they want, how they want without limits.

To begin the process, we met with several great Drupal design teams last fall. One, Four Kitchens out of Austin, told us about an exciting new technique they had developed for a couple of their big media clients. This technique, called "headless Drupal," used the robust Drupal content management system for data entry (I'm writing this in a Drupal form right now) and to serve the API calls. For the web front-end they recommended a more modern and flexible tool called node.js. Unlike previous web sites, the site itself wouldn't be a content management system, it would only be a representation of the content, created by calling the API. I liked the idea for several reasons.

First, Drupal's presentation is looking a little old and staid next to the spiffy, responsive sites other media companies are using. I liked the idea of being able to use a more modern tool like node.js.
Second, by decoupling the backend from the presentation, it would be easier and faster to create new sites. Web design styles change faster than high fashion, so it's nice to be able to update the site without re-doing all the hard work on the backend.
Third, having a complete API would make it easier to do apps. The app, just like the website, would have access to everything there is to know about TWiT, in a simple, accessible fashion.
Fourth, by making the API public, we encourage members of our audience to create new things, things we might never have thought of. You could even design a website you like better. Abstracting the content from the presentation seems like a big win.
Finally, by keeping Drupal simple and avoiding additional third-party modules, we can make a more secure and reliable backend that will be much easier to upgrade when future versions of Drupal arrive.

So while you may think we're unveiling a new website here, we're really just showing you the tip of the iceberg. Most of the work was done on the backend and is invisible to you but front and center for our team.

Design Decisions

We asked the Four Kitchens design team to create something very visual and clean for the new TWiT.tv. We feel that people come to the website for two reasons: first because they'd Googled one of our hosts or were searching for tech information. Those people need to get a quick idea of what TWiT is and what we do. The front page is designed to do that, with big hero images, quick statements of purpose, and a gallery of recent shows in our three main categories: tech news, help and how-tos, and reviews.

The second reason people come to the site is to find content. These people already know about TWiT, but they want to quickly find and download a show, watch the live stream, or search for information they heard on a show. The top navigation reflects those primary purposes: Live, Shows, Apps, Search and More... (where we hide everything else). We're using Apache Solr search and I think it's a major upgrade to our old search.

Most importantly the new site had to look good on any size screen. You can see for yourself by resizing the page and refreshing. We think the site might even look best on mobile and that's good because that's how most people visit us these days.

Credits

My deepest thanks to our Four Kitchens designers and engineers: Caris Hurd for interface design and layout, Jared John, for the look and feel. Matt Grill was the architect of the node.js and dust apps and lead implementer on the node.js side. David Diers was our kindly Drupal wizard and API designer. Peter Sieg played all-star utility guy moving between node.js and Drupal and catching the hail marys that Matt and David threw his way. Kevin Lamping also worked on the dust templates - if this site looks good on your mobile device you can thank him and Jared.

The entire process used the Agile methodology, and our project leaders, Paul Benjamin and Suzy Bates, kept the Jira backlog humming through 279 user stories and 730 developer points in nine sprints over the past six months. And if you know Agile you'll know what that means.

The whole Four Kitchens team worked miracles, and if you enjoy the site, buy them a beer the next time you're in Austin. I want to buy them a whole brewery.

Any missing or broken features are not their fault - they're ours. We had a strict budget and weren't able to add many of the cool features we all wanted. But we have an ongoing support contract with Four Kitchens and hope to add the most requested missing features over time. We have a backlog of well over 100 features we didn't have the time or money to implement in version 1.0. I suspect we'll be ready to do a second phase with Four Kitchens when everybody recovers from this one. In about a year.

On the TWiT Team, credit to the quiet but mighty, Patrick Delahanty, who re-coded our workflow engine, Elroy, to interface with Drupal so that our editors and producers could move their entire data entry workflow to the Drupal backend. And he did it in about a month. Patrick is the official first API user. I managed the project with help from our CEO, Lisa Laporte, Engineering Manager, Bruce Chezem, and our peripatetic IT consultant, Russell Tammany.

The Drupal 7.0 backend is running on B lackMesh servers using memcache and Varnish to speed things up. The web front end is running on Heroku. The API is documented at Apiary with key service from 3Scale. Node.js caching is provided by RedisLabs. Video and audio playback on the site is from JWPlayer. All TWiT shows and RSS feeds are stored and served by CacheFly.

Many of the pictures you see on the site were shot by Jason Guy at Jason Guy Photography.

I hope you like TWiT.tv 4.0. Please give me your feedback at leo@leoville.com. We'll fix the bugs as fast as we can, and adjust the interface to better suit your needs when we can afford it. Meanwhile, brush up on your RESTful API calls. I can't wait to see what you do with it.

For more information about our developer program and the public API visit twit.tv/about/developer-program

Leo Laporte, 12 June 2015