Blog for The Well Designed URLs Initiative

As a way to gather a broad spectrum of opinion and engage the community on many different URL-related topics, we’ll be offering a URLQuiz from time to time. Inspired by and patterned after Dan Cederholm’s very well received SimpleQuiz series, we’ll take a simple URL Design question, present a few different alternate approaches, and ask readers to weigh in on which they believe to be the best and why.

Dan proved that his SimpleQuiz was very effective at quickly emerging a consensus for a best practice when a consensus was possible. Building on Dan’s pioneering efforts, we hope to leverage his technique to address the constantly debated issues in URL Design and hopefully arrive at some consensuses of opinion on URL-relates issues ourselves.

So look for the first URLQuiz, part of an ongoing series, here at the Well Designed URLs Initiative blog in the near future.

For our web applications — our website, wiki, blog, and future forums — we chose to segment each by sub-domain; “wiki.”, “blog.”, and “forums.”, respectively. Rather than having subdirectories underneath the root of our www domain such as “/wiki/”, “/blog/”, and “/forums/”, we chose to use sub-domains to allow flexibility and maybe even scalability in the future.

Having the each application on a domain that is different from the blog and forums mean we can host on different machines and at different locations, as appropriate. However as I write this I realize that this rationale violates probably one of the most important best practices for URL Design; don’t allow your backend implementation dictate your URL structure!

It also means our URLs will be by 4 characters shorter[1] and since we chose a long domain name, that’s important. Still, there are pros and cons to both approaches and my gut tells me we’ll uncover further considerations as we continue researching this topic.

Related to this topic, we’ll discuss the URL Structure patterns for websites, wikis, blogs, forums, and other web apps in future posts.

1. Assuming we were to continue using “www.” as a sub-domain for our main site, which itself is another topic for us to address in the future.

Here’s a run down of the web application software we are using or planning to use for the Well Designed URLs Initiative:

Blog

At “blog.” is running WordPress using Linux, Apache, MySQL, and PHP. We have direct experience with WordPress, Typepad, dasBlog, and Community Server, and among these options WordPress is definitely the best blog software for the URL Aficionado.

Wiki

At “wiki.” runs Mediawiki, also on using Linux, Apache, MySQL, and PHP, and is the same software used by as Wikipedia. Although setting up clean URLs can be a real pain on Mediawiki if you are not a regular expressions wizard, we think Mediawiki is the most mature wiki software, and nobody can debate that it has the best known user-interface.

Forums

To be at “forums.” will almost certainly run on vBulletin which also runs on Linux, Apache, MySQL, and PHP. Even though vBulletin isn’t open-source and costs US$85/year to lease[1], as users we have used practically every forum in existence and in our opinion nothing comes close to the usability and performance of vBulletin. We only wish that the vendor of vBulletin could envision a business model leveraging open-source so those who for whatever reason won’t use commercial software would be able to consider vBulletin; we think that would make the forum world a nicer place. Well that, and changing their default template; the visual layout as shipped is horrible!

CMS

To be at “www.“; we’ll probably use Drupal, Django, or similar when we get to installing and configuring it. In other words, when we really need the features of a CMS, we’ll add one. ;-)

The all the web applications we mentioned are great, and we can highly recommend them to almost anyone. The downside of using them though is being separate apps they are not well integrated. But, we’ll see what we can do about that in the future. Hopefully someone will see what a huge need and certain demand for integrating open-source web apps based on MySQL and they will solve this problem just as the WSGI team seems to be solving a similar problem in the Python web app world.

1. Or you can purchase vBulletin for US$160 with one year of updates.

One of our plans is look at major topics from many different angles. Though this will allow us to revisit an issue numerous times, our main reason for covering topics in this manner is that making sure our coverage is complete and includes all references is really difficult. And we (well at least, I) have a tendency to postpone writing on a topic until I’m sure my research is exhaustive. Unfortunately, that often leads to me writing about something long after it’s relevant!

So rather then delay an important or timely topic, the plan is to both write about them numerous times with each time taking a slightly perspective and references different resources. And we plan to publish those multiple angles in the chronological of our normal blog publishing cycle.

But we also plan to maintain a section of Major URL-related Topics by consolidating the content from our numerous chronological posts on a topic and often augmenting with additional content not published during our normal blog cycle. You’ll find those pages at the following URL:

http://blog.welldesignedurls.org/major-topics/

Unlike our category pages that are generated by a SQL query from our prior posts by WordPress, we’ll actually be hand writing the content for these major topic pages. And we’ll do our best to keep those pages updated to cover our most recent research and views on the subject. In addition, we’ll reference any of our related posts and also provide quotes and links from related conversations on the Internet including those from other bloggers, mailing list archives, and online magazines.

Our goal is to make these major URL-related topic pages your most thorough, up-to-date, and hopefully concise resource on the subject. And we hope you’ll link to them if you need to reference one of our major URL-related topics when writing an article or posts for your own blog.

NOTE: To see why this is labeled “Part 1a” see [1].

Although many people view the URL to be a rather obscure little creature, the URL is the arguably the most important of the web’s three essential legs; the “First Consul” of the triumvirate of web technologies; the truly unsung hero of the web.

When Tim Berners-Lee originally designed the Web, he did so by creating three core independently-specified yet inter-related technologies on top of the existing TCP/IP-based Internet. Used together, those three technologies leveraged each other to create spectacular value for the world.

The first of those three technologies, HTML, has been broadly appreciated by web developers and even lately a tremendous number of end-users for allowing them to layout text and graphics how they prefer them to be display[2]. The second, HTTP, is intimately familiar to most web users, but only as the prefix they see in countless television ads, on business cards and billboards[3], and in their web browser’s URL field as they use hypertext to navigate the web.

But whereas both HTML and HTTP are independent of the other, both rely heavily on the URL. Without the URL, HTML would have no Hypertext and HTTP could neither retrieve documents[4] from websites nor issue redirects when documents move. Berners-Lee even “canonized” the URL[5] in his 1999 Wired Interview:

“Well, the most important thing that was new was the idea of URI — or URL [it was UDI back then, universal document identifier]. The idea that any piece of information anywhere should have an identifier, which will not only identify it, but allow you to get hold of it. That idea was the basic clue to the universality of the Web. That was the only thing I insisted upon.”

But like the man behind the curtain to whom we are told to pay no attention in the Wizard of Oz, many web technologists have mistakenly promoted the notion that the URL should only be used for behind-the-scenes plumbing and thus hidden from view. Sadly, these technologists rationalize their stand by stating that mere-mortals couldn’t possibly understand URLs anyway, but nothing could be further from the truth. For this reason and more, URLs true potential has gone unfulfilled as they have been woefully under-appreciated in comparison to the power and benefits they can provide both the average web user and the web at large.

The URL, a.k.a. the URI, is more than just some behind-the-scenes, don’t-speak-until-spoken to technology. URLs provide a means to identify both concepts and tangible things and then allow related documents to be retrieved that can provide more information and/or further reference URLs. While most people think in terms of how they personally use URLs, they rarely envision the orders of magnitude of serendipitous uses by others of the URLs they assign and then publish.

It’s often said that what you don’t know can’t hurt you, but unfortunately this lack of understanding the power of the URL creates huge missed opportunity for most people. Just one simple example should illustrate this and that is the value of Google’s search engine indexer; without URLs there is no chance Google will send anyone to learn more about a concept or tangible thing that could have been made accessible by URL but wasn’t. Seeing how much money Google makes from advertisers seeking traffic to their website, it’s amazing that more people don’t realize the value of the indexable URL. But Google’s use of URLs is the mere tip of the iceberg when compared to all the serendipitous use one can receive by assigning and publishing Well Designed URLs for everything they view should be notable to others.

Tim Berners-Lee has been quoted as saying “Everything of importance deserves a URL“[6] and it is our opinion that is probably his most profound statement ever. So one of the key goals of the Well Designed URLs Initiative is to right the wrong of past technologists who have been hiding URLs from the average user’s view. By elevating URL’s the significance in the minds of both Internet professionals and the masses we hope to be the catalyst for the positive change that the recognition of URL’s value will bring.

1. This part of our introduction was published out of order hence it’s designation as “1a” instead of as “2.” During the process of writing it became clear that this needed to be said but unfortunately we had already published numerous posts that would have obviously come after this one. Ah well, that’s what happens when you start publishing before you finalize the outline. :-)
2. Graphic designers would argue this point, but that would be nit picking for the purposes of this blog post.
3. Albeit the trend is toward omitting and hence assuming the “http” written and spoken communication.
4. The Weborati would correct me and say that HTTP retrieves “resources“, but why quibble over common use when this post is non-normative?
5. My use of the term “canonized” is obviously just a bit of “dramatic license.”
6. Actually Tim uses URI instead of URL which is his preference vs. mine. Go here to read about URL vs. URI here.

Here at the Well Designed URLs Initiative we plan to address a wide audience and cover a plethora of URL-related topics. If it wasn’t obvious from yesterday’s post we plan to publish content for a variety of roles so we will categorize all our posts by the audience we are targeting.

Using our audience categories you can subscribe to our RSS feed then configure your feed reader to filter out all but those topics which are likely to appeal to you so as not to be overwhelmed by the rest. Some of our audience categories encompass other categories such as Everyone and Internet Professionals so we’ll plan to tag only the highest level category that applies to avoid duplication. For example, if you are a web developer you might want to filter out all but the Everyone and Internet Professionals, and Web Developers categories. Of course, if that’s too much trouble just subscribe to our entire feed and just ignore those posts that don’t interest you.

The following is the list of categories we’ve set up by audience role:

• All Internet Professionals
• Conference Promoters
• Teachers
• Everyone
• Framework Developers
• Hosting Providers
• Infrastructure Providers
• Leading Companies
• Open-Source Participants
• Content Publishers
• SEO Consultants
• Software Vendors
• Standards Participants
• Web App Providers
• Web Designers
• Web Developers
• Website Owners

NOTE: If you read this post shortly after it is published, most of those links above will just redisplay this post. Let me explain. Because of the way our WordPress blog software works, those links would have displayed a 404 Not Found error if no posts existed for the given category. To avoid that I’ve tagged this post with all audience categories contradicting what I said above; that we would only put a post in its highest level category. Moving forward we shouldn’t need to do this again.

We’ve already said that we plan to question the status quo, but we want to make that point even stronger; we plan to advocate for positive change.

At least one person has already suggested that our activities would be best accepted if we, like the microformat initiative aimed to merely “uncover what already works” on the web and then dispassionately documented it for discussion. Although we do plan to research and then explain current patterns and best practices we don’t plan to make those our only activities; we didn’t pursue this merely to be the scribes of current behavior.

Instead, we are promoting this initiative because we envision specific solutions to current URL usage problems, and we believe those solutions will significantly improve both the web’s accessibility and user’s experience with the web. It is this vision that motivates us to pursue this path even if it means that at times it might be an uphill battle.

But what specifically do we plan to advocate? To give you an idea I prepared the following chart, organized by who will be the target of our advocacy:

Although this list of objectives is significant, it doesn’t represent the entirety of what we hope to accomplish via this initiative. Stay tuned.

Yesterday we said we plan to make recommendations. That may have concerned some, especially those who are involved in the standards process. But don’t worry, we won’t make them in a vacuum; we intend to engage everyone interested to participate and influence the creation of the recommendations, especially the leading experts in the field of web architecture. We pledge to evaluate everyone’s input objectively, and incorporate input without prejudice.

What’s more, we aim to respect all the standards created by recognized standard’s bodies such as the Internet Engineering Task Force and the World Wide Web Consortium. Further, we plan to respect the spirit of all standards and acknowledged axioms.

Though we did expose a long list of agenda items, some of those may fall by the wayside once we engage the community. Those were our ideas but as we are not perfect some of them may not be either. Our goal is to make things better, not worse, and intellectual honesty is our only ally in addressing reality.

But that doesn’t mean we intend to always be non-controversial. No, we intend to push the envelope and lead some people out of their comfort zone. Like a child constantly asking “Why?” we plan to challenge conventional wisdom. We want to ensure that the axioms and standards defining web architecture are still based on actual requirements. We don’t want to see progress impeded simply because the framers of the web shared the universal human trait of being usable to foresee all future innovations and their requirements.

Though potentially disruptive, we think our willingness to question prior assumptions will ultimately lead to a better web for all.

One of our main goals is to publish an evolving list of URL Design and related recommendations that the world of web architects, website administrators and web developers will respect and embrace. Our recommendations will include advocacy for, but not limited to the following:

• Establishing Sitemaps as fundamental in URL Design,
• Recognizing URL Patterns as first class concepts,
• Conventions for file-extension to media-type mapping,
• Deprecating Regular Expressions for URL Rewriting,
• Use of URI Templates where applicable,
• Hardening URI Templates for real world usages,
• URL-related Metadata in HTML and HTTP,
• Formalizing the use of Canonical URLs as identifiers,
• Exposing Canonical URLs to users,
• Leveraging the Permalink to identify Canonical URLs and more,
• Handling URLs with Embedded Session Identifiers,
• Handling URLs with Embedded User Identifiers,
• Handling URLs with Embedded Print Modes,
• URL Virtualization to overcome platform limitations,
• Well-known Media-Types for REST’s Hypermedia Constraint,
• Making URIs Dereferencable whenever possible,
• Strategies for dereferencing every URI to a resource,
• Creating Unique URLs for all Dereferenceable Resources,
• Rationalizing Content Negotiation vs. Unique URLs,
• A Web Request Media-type for POST, PUT, and DELETE,
• Strategies for serving Alternate Language Representations,
• Making AJAX, Flash, and PDF resources dereferenceable,
• Publishing Multiple URL Paths to the same resource,
• Metadata to allow greater URL equivalence evaluation,
• Recommending software based on URL configurability,
• Recommending  to vendors how to improve URL configurability,
• Exposing Segmentable Content via URL Fragments,
• Exposing URLs for all deferencable content,
• Strategies for handling Broken Links,
• Strategies for URL-related Education for all users,
• And even more.

So you see, we’re planning to bite off an awful lot here but we think by doing so we’ll benefit all web users. Didn’t realize that URLs had so much to room for improvement? Stay tuned…

There is a belief among the people who guide the web’s technical architecture, people I like to call the “Weborati“, that as few constraints as possible should be placed on web publishers and web users. This is referred to as the Principle of Minimal Constraint. An example of such a constraint is the Robots Exclusion Standard that uses the “robots.txt” filename to inform search engines of pages to avoid. Because of this robots.txt “standard”, a web publisher can’t use a file named robots.txt in the root of their website for any other purpose besides the Robots Exclusion Standard without disabling their ability to inform web robots which files to ignore.

Although using the Well-known Name “robots.txt” was justified by Roy T. Fielding, use of well-known names such as those required by P3P and Favicon continue to be a sore point among the Weborati. However, contrary to some initial concerns, we here at the Well Designed URLs Initiative agree that using well-known names is A Bad Thing(tm) in Web Architecture. So let me state for the record: It is not our intention to encourage any new well-known names.

On the other hand, we do intend to recommend convention. Good conventions can increase productivity immensely by eliminating the need to evaluate alternate yet arbitrary choices. One such convention is the multi-column web page layout complete with a menu along either the top or one of the side columns. Prior to that established layout, right for many websites but not all, web designers spent considerable time evaluating different layouts. I, for one, am glad to have gotten past that stage.

Good conventions can aid common understanding if those conventions are widely used. For example the Internet sub-domain naming convention of “www” has helped many non-technical people recognize a web address on a business card, a TV advertisement, and on the side of a bus.

And good conventions can spawn an entire industry segment, unleashing a completely new layer of value. When Jesse James Garrett published his now well-known essay entitled “Ajax: A New Approach to Web Applications,” he recognized a design pattern that many people had been using in obscurity, and that recognition catalyzed a mega-million dollar segment of the web development industry.

Though we could only hope our efforts have the same effect, we do hope to influence how people perceive and use the lowly URL. For example via research and a consensus of opinion, it would be greeat to establish a widely used convention for how multi-lingual websites design their URLs.

Although conventions can be constraining, conventions are not in and of themselves requirements. Just as it is well known that “www” probably means “website”, it is not a requirements and the fact many websites use alternate sub-domain names prove it. For example, “newyork,” “london,” and “paris” could all easily be used as sub-domain names for relevant websites. Or a website can easily use no sub-domain at all!

So it is our hope we can get people from the Weborati down to the non-technical website owners to appreciate and embrace numerous new URL conventions in the future.

So what makes us “the” expert on URL Design as we are not (yet?) associated with any standards body like the W3C or the IETF? Nothing really, other than we have a keen interest in the URL Design and its ramifications, and are proactively devoting time to this initiative.

For me personally, I’ve spent lots of time spent studying any and every bit of writing I could find regarding URL Design, and even more time spent pondering the subject and writing about it. Lastly, it was my writing about “Well Designed URLs” and other’s subsequent interest that lead me to start this initiative.

Plainly put, I and the others involved are the first to take significant initiative to study and then advocate for User-Centered URL Design. We feel so strongly about this that we believe it should be taught in universities as part of the new field of “Web Science.”

As with any body of wisdom it has to start somewhere, and because no one else was focusing on URL Design, we decided we would start I here; it’s that simple.

That said, if you feel that you have something significant to contribute to URL Design then by all means join us and become one of the experts yourself. And if we do a good job, we may become recognized by the W3C and/or the IETF, and that will make us the experts. :)

To this point, I have mostly written in the collective as opposed to the personal; “we” instead of “I.” However, right now “we” is officially “me.” Several others have signing on to indicate they believe in our mission, and I’ve discussed concepts directly with at least one other person, but at the moment no one has “officially” joined me in this initiative.

However, I hope to revise the situation in the near future so that “we” is officially many people. I’d love to have some company here; I didn’t start this project to be “all about me.” Instead I hope to leverage the research and experience of the many smart people who have contributed to the web in one form or another over the more than a decade of its existence. If all goes well, there will be more than just me blogging here about URL design before long.

So I write in the collective because I expect that like-minded people will join me in this initiative, and by self-selection they will identify with what I’ve published thus far. Since I expect this introduction to live on for a long time, ultimately “we” will refer in a retroactive sense to all of those involved, not just me.  That’s my hope, at least. :)

Many people see URLs and think only of technical concerns. You can see this thinking in websites implemented using Vignette and in most websites implemented using Microsoft’s ASP.NET. This we feel this shortsightedness is a serious error in judgment.

Alternately, we believe that the study of URL Design should be an important component of Web Science. And, like Web Science, we believe URL Design must consider many factors to truly optimize Uniform Resource Locator use for both man and machine. Those factors that URL designers really should consider minimally include the following:

• Cognition,
• Culture,
• Business process,
• Law,
• Economics,
• Politics,
• Sociology, and finally
• Technology

Clearly URL Design is not just about mapping files and scripts on a server.

First and foremost, the focus of the Well Designed URL Initiative is User-Centered URL Design. Although URLs a.k.a. URIs are used in many places that typical users never see, our primary focus will be to advocate for the designing of those URLs that users do see.

And by “users” I don’t just mean people who are surfing the web using a browser but also people who interact with URLs on many other levels. Those other users include the web developers who build and maintain websites as well as the system administrators who install and maintain the web servers. These users need URL usability just as much as average users, if not more.

What we don’t plan to address, necessarily, are those URLs that humans never see. Yes there are a lot of systems that generate URLs and pass them around from computer to computer but those are not our focus here. The vast majority of URLS are seen by humans[1] and as such are assertion is that those URLs should be understandable by humans too.

At least that’s our contention.

1. Even when URLs are machine-generated and used only by machines to communicate with other machines humans still need to debug those systems, so I could argue that well designed URLs make sense even there. But I won’t split hairs today.

At first glance, you might think that the Disciple of URL Design is a relatively narrow subject. Au Contraire my good reader, the subject of URL Design is miles deep and oceans wide. URL Design permeates practically everything about software and the web today. Almost every web standard that’s been recommended or even discussed as well as almost every recently-released software package leverages the lowly URL.

So be aware that URL Design extends mightily into almost all aspects of web and software design, and as such studying URL Design, understanding what constitutes a good URL design, and defining URL Design Best Practices can bring great benefits for all.

Hello and welcome to the first of a series of introduction posts for the Well Designed URLs Initiative blog. During the next month I plan to present why I launched the Well Designed URLs Initiative, provide background on URLs and their history, and present my strategy and goals for the initiative. I’ll be posting each day Monday through Friday until I’ve posted the entire introduction. I’ll also include a list of links to all the introduction posts here.

After that, I’ll get on with the real business of the initiative and this blog; advocating for User-Centered URL Design.

If you are checking out this blog and dismayed that we’ve got no content online yet, please note we’ll currently doing a ton of research on good Url design so we won’t post uninformed opinons and then look like ignoramouses!

But please be sure to subscribe your RSS reader because we’ll soon be publishing a plethora of information about good Url design.