Archive for the ‘Technologies’ Category

URL Quote #4: An utter disaster to disable the URL

Sunday, September 23rd, 2007
“Given that social filtering is one of the most powerful mechanisms for information discovery on the Internet, it is an utter disaster to disable the URL as an addressing mechanism. “

-Jakob Nielsen on “Why Frames Suck (Most of the Time)

Proposing HTTP Request Forwarding

Thursday, March 8th, 2007

I’ve been monitoring the ietf-http-wg mailing list and have noticed there is renewed actively around revising RFC 2616, the HTTP/1.1 specification. This renewed activity got me thinking it was time to discuss the need for HTTP Request Forwarding.

I’ll start by saying it is possible that Request Forwarding is already part of the HTTP spec and that I just overlooked it. Lord knows I’ve read enough W3C and IETF specs lately to raze a small forest if printed, so I could easily have missed that part as I read bleary-eyed through the specs. But I’ve asked the question privately of a few people that should know and they all said that HTTP does not support request forwarding; one of them pointed out that that is why VOIP needed SIP.

If the term “HTTP Request Forwarding” isn’t obvious from context, let me give a use-case to illustrate:

USE-CASE: Retaining control of URL Interfaces when outsourcing image hosting

  1. http://example.com
  2. http://image-servers-r-us.net
  3. http://example.com/index.html
  4. http://example.com/images/splash.png
  5. http://image-servers-r-us.net/example.com/splash.png

Assume client “A”, server “B” mapped to [1] and server “C” mapped to [2] (servers “B” and “C” are different computers with different IP addresses probably at different locations.) For this use case client “A” requests the HTML file at [3] which includes an <img> tag pointing to a graphic at [4].

The request from client “A” for the .html file at [3] goes to server “B” and the response is returne to client “A.” After parsing [3] client “A” realizes it also needs to download the image at [4] and requests the .png file from server “B” at [1].

HTTP Request Forwarding (Note use of HTTP 1.2 to make URL in example explicit)

However, server “B” knows that the .png file [3] is actually located on server “C” at [5] so it forwards the request to server “C” at [2]. Server “C” responds by returning the .png file directly back to client “A” and client “A” is none the wiser; i.e. client “A” still thinks that the image was returned by [4].

In addition, if client “A” is a browser and the user inspects the properties of the image, the user would find the URL to be [4], not [5]. No where in the response given to the client would there be information that the image came from [5] instead of [4] except that the IP address differs from the server that requested it. It is possible that a header could contain the forwarding information, but the client would not need to act on it except potentially for debugging.

There are several other use-cases as well, such as making it possible to fully virtualize a domain authority’s URL interface. However, the use-case of outsourcing images or any large static content illustrates a benefit that I believe should be obvious to most people. And for those that are interested, especially if you are involved in updating the spec, I could detail other use-cases if people are interested.

Now it is possible there are security implications with this that I have not considered. If so, I would hope that we could at least explore potential ways to mitigate those issues as opposed to immediately dismissing HTTP Request Forwarding out of hand.

In summary, adding a Request Forwarding functionality to HTTP would allow servers to maintain control of their URL interfaces while still being able to distribute loads and services to appropriate servers. Having such a feature could improve consistency in URL design and make it easier for website owners to restructure their websites without the level of broken links seen on the web today.

Interestingly, on Tuesday Scott Hanselman blogged about just this problem from a slightly different perspective; I present Scott’s post for your consideration.

URL Quote #3: Wikipedia’s URLs a reason for their success?

Wednesday, March 7th, 2007
“Wikipedia’s URL spaces are highly elegant; I suspect it’s one of the reason’s Wikipdedia is successful”

-Bill de hOra on “Wikipedia’s Highly Elegant URLs

URL Quote #2: Think about your website’s “public face.”

Saturday, March 3rd, 2007
“…one should take an hour or so and really think about their website’s ‘public face.’”

-Scott Hanselman on “A Website’s Public Face

URLQuiz #2: URL Equivalence and Cachability

Thursday, March 1st, 2007

This is quiz #2 of our ongoing URLQuiz series.

In this quiz, there are 26 pairs of URLs (A..Z) and for each pair the questions is: “Which of these two URLs are equivlent?” i.e. which return the same resource when dereferenced, and “Which can be cached as the same URL?

You should answer ‘Yes‘, ‘No‘, or ‘Maybe‘ where ‘Maybe‘ means ‘It might return the same resource but should not be cached according to the specs.’

To answer, leave a comment and ideally explain your reasoning for each. Feel free to group answers based on your reasoning and/or the answer given (Yes, No, and Maybe.) Print it out and take the quiz with pencil and paper if you serious about getting it right, and feel free to use a computer or browser or whatever to test your results before answering. Good luck!

About the TLD .foo[1]

Clarification (2007-March-02): Some people have stated that the server could possibly return the same resource for any two given URLs so they felt the answer could never be ‘No.’ I definitely see their point, for example http://mysite.foo/bar and http://mysite.foo/bazz could possibly return the same thing but nobody would ever reasonably expect them to do so on their on. So let me clarify to say that I meant a quiz taker to select ‘No‘ in the case where where the resource returned would definitely not be the same thing unless the developer or server admin explicity programmed or configured them to do so. On the other hand ‘Maybe‘ would be used in the case where someone might reasonably expect the two URLs to return the same resource even though the RFC 3986 would define the two URLs as being different such as in [footnote 2], or when it depends on the O/S of the server as in [footnote 3]. Regarding fragments, the question is “In a transaction between a client and a server, is the cache allowed to view them as the same?” Regarding which can be cached I was looking for what are appropriate per the spec, not necessarily whether any particular software in the cloud (i.e. routers, proxies, browsers, etc.) actually does cache but instead “Would it be allowed to cache?”

Questions

  1. The ‘www’ domain
    1. http://mysite.foo/
    2. http://www.mysite.foo/
  2. Letter casing in path
    1. http://mysite.foo/Index.htm
    2. http://mysite.foo/index.htm
  3. Letter casing in domain
    1. http://MySite.foo/index.htm
    2. http://mysite.foo/index.htm
  4. Index.htm vs. Default.aspx
    1. http://mysite.foo/Index.htm
    2. http://mysite.foo/Default.aspx
  5. Trailing slash on domain
    1. http://mysite.foo
    2. http://mysite.foo/
  6. Trailing slash on path
    1. http://mysite.foo/path
    2. http://mysite.foo/path/
  7. Empty question mark
    1. http://mysite.foo/
    2. http://mysite.foo/?
  8. Empty parameter
    1. http://mysite.foo/?
    2. http://mysite.foo/?param=
  9. Port 80
    1. http://mysite.foo/
    2. http://mysite.foo:80/
  10. Port 443
    1. http://mysite.foo/
    2. http://mysite.foo:443/
  11. Https vs. Port 443
    1. https://mysite.foo/
    2. http://mysite.foo:443/
  12. Ftp vs. Http
    1. ftp://mysite.foo/
    2. http://mysite.foo/
  13. Letter casing in parameter name
    1. http://mysite.foo/?param=bar
    2. http://mysite.foo/?Param=bar
  14. Letter casing in parameter value
    1. http://mysite.foo/?param=bar
    2. http://mysite.foo/?param=Bar
  15. Hash vs. no hash
    1. http://mysite.foo
    2. http://mysite.foo#
  16. Hash vs. Fragment
    1. http://mysite.foo#frag
    2. http://mysite.foo#
  17. Fragment vs. no Fragment
    1. http://mysite.foo#frag
    2. http://mysite.foo
  18. Plus vs. Space in path
    1. http://mysite.foo/url+design
    2. http://mysite.foo/url design
  19. Space vs. Encoded Space in path
    1. http://mysite.foo/url design
    2. http://mysite.foo/url%20design
  20. Plus vs. Encoded Plus in path
    1. http://mysite.foo/url+design
    2. http://mysite.foo/url%2Bdesign
  21. Slash vs. Encoded Slash in path
    1. http://mysite.foo/top/second
    2. http://mysite.foo/top%2Fsecond
  22. Ampersand vs. Encoded Ampersand in path
    1. http://mysite.foo/abc&xyz
    2. http://mysite.foo/abc%26xyz
  23. Ampersand vs. Encoded Ampersand in parameter value
    1. http://mysite.foo/?q=abc&xyz
    2. http://mysite.foo/?q=abc%26xyz
  24. Equals vs. Encoded Equals in path
    1. http://mysite.foo/abc=xyz/
    2. http://mysite.foo/abc%3Dyxz/
  25. Equals vs. Encoded Equals in parameter value
    1. http://mysite.foo/?q=abc=xyz
    2. http://mysite.foo/?q=abc%3Dyxz
  26. Parameter order
    1. http://mysite.foo/?abc=123&xyz=987
    2. http://mysite.foo/?xyz=987&abc=123

P.S. Don’t stress if you can’t answer them all. It took me months to uncover all these nuances, and if I were taking this quiz I doubt I could get them right all in one sitting.

FootNotes

  1. I’m using the non-existent top-level domain “.foo” to avoid giving any link-love to arbitrary example sites that don’t deserve it! For the purpose of the quiz, just assume that “.foo” is a functioning top level domain.
  2. Question A.
  3. Question B.

Sorry Mark; URL Design DOES matter!

Wednesday, February 28th, 2007

I was planning to blog something else today, but Mark Nottingham of Yahoo made a statement about URL Design [1] in his post entitled REST Issues, Real and Imagined and I simply could not let his statement without comment.

But first let me say I always appreciate Mark’s perspective on issues, and enjoy reading his writings whether on his blog or in the mailing lists. His perspective is typically insightful and prescient, and he hovers above the hovers above the muck-racking and unsupported claims that can occur on mailing lists populated by egos. All in all his involvement is very professional, and I highly respect him for that,

That said, here is the comment me made that really bothered me (2nd and subsequent emphasis mine):

Red Herring: URI Design
When somebody first “gets” REST, they often spend an inordinate amount of time agonising over the exact design of the URIs in their application; take a look over on
rest-discuss, for example. In the end, though, URI design is a mostly a cosmetic issue; sure, it’s evidence that you’ve thought about good resource modelling, and it makes things more human-intelligable, but it’s seldom worth spending so much time on it.

I’d worry a lot more about cacheability, extensibility and well-defined formats before blowing out my schedule on well-designed URIs. For me, the high points are broadly exploiting the hierarchy, allowing relative references, and making sure that tools (e.g., HTML forms) can work well with my URIs; everything else is gravy.

I’ll start by saying I don’t really disagree significantly with his overall points that I believe he was making, such as the fact that there are other aspects that deserve attention in addition to URL design and also the fact some people appear to thrash when designing URLs. But to someone who does not appreciate URL design his statement could be easily misconstrued to mean that URI Design is not at all important, especially when he says it is mostly a cosmetic issue. That is false.

URI design is NOT merely a cosmetic issues! It has many tangible ramifications and those who ignore it do so at their own peril. Just scratching the surface, proper URL planning and design provides a framework for good information architecture, can facilitate spontaneous inbound linking via blogs, voting, tagging, and other social media sites, and can guard against broken URLs and subsequent traffic loss, to name but a few. They are issues that concern, or should concern everyone who publishes a site on the world wide web.

But what’s worries me most are the people who will certain latch onto Mark’s words as not only justification for ignoring patterns and best practices but also for antagonistically preaching against URL Design. This group includes web and content management system developers who would prefer not to be bothered with usability issues, system administrators who believe in security by obscurity (which itself is a fool’s precaution), dogmatists who misinterpret the principle of URI opacity and preach that web publishers should publish completely opaque URLs,  opaque even to the web publishers themselves, and a tiny but vocal contingent that for reasons I cannot fathom argue against URI design even within totally unrelated conversations.  It is for this reason I think Mark’s statement is potentially very damaging.

And as for his comments about rest-discuss, it is quite possible he was referring to conversations in which I participated. If so, I believe he mistook the crux of the conversation on several levels. The first was that in many cases I was advocating URL design, not agonizing over it. Certainly, URI design is really not that hard, it just takes understanding core principles and best practices, and then applying them. And secondly, some people escalate conversations to raging debates when simple questions are asked about proper URL usage in context of REST and Web Architecture. Mark could easily yet wrongly have misinterpreted those as too much hand-ringing over URL design.

In summary I don’t really fault Mark’s comments as I appreciate them to be. And I think Mark does a great service for the web in his quest to educate people about the value of REST. But as Mark is justifiably well respected I’m worried Mark’s comments may be used to rationalize bad URL Design. As such, I definitely hope he updates his post to guard against his word’s misuse.

Footnotes

  1. Mark prefers to use the term URI instead of URL whereas I obviously prefer the later when in the appropriate context. By definition, URLs are URIs with the difference being that a URL is dereferenceable. When discussing REST end-points, the term URL is applicable and the term URI which refers to non-derefenceable identifiers, is not. And don’t you forget it! ;-)

P.S. Oh, and I also couldn’t help but wonder if Mark was trying to get in a cheap dig when he wrote “…before blowing out my schedule on well-designed URIs“…?  But naaaaah, Mark’s a real professional and wouldn’t go for such an underhanded shot. ;-)

PageRank

Monday, February 26th, 2007

SEO: Illuminating the value of URL design

If you’ve read many of our other posts here at The Well Designed URLs Initiative, you know that we are strong advocates for User-Centered URL Design as well as for URL Literacy. It’s our contention that the URL is woefully under-appreciated as the most fundamentally important technology of the web, more important than HTTP, and even more important than HTML. The purpose of this post is to provide background for future posts explaining URL design importance from a perspective most website owners can appreciate; search engine rankings!

But they’d have to kill you

For those unfamiliar with Google’s core algorithm for determining its search engine results, you can read this article to learn about PageRank in more painstaking detail. Here, I’ll just try to explaining the aspects of PageRank as it relates to URLs. Note also that my explanations are simply meant to be a conceptual guide and not exacting details. The founders of Google did publish their initial algorithms but have since made tweaks that are as closely held a corporate secret as the formula for Classic Coke!

Popularity is the key

PageRank ias essentially a popularity rating, and a page’s PageRank is determined by the inbound links from other pages on the web. A PageRank can be as low of almost zero (0) to as high as ten (10). Google’s algorithms determine a page’s PageRank by dividing the PageRank for each of the inbound linking pages by the number of outbound links on each of those pages, factoring in each page’s PageRank, and then summing the results for all inbound links. Clear as mud, right? It’s easier to explain with an example, but let’s cover a bit more ground first.

Like voting company shares

PageRank considers each link a ‘vote’ for the page linked to. But unlike in a democratic “one citizen, one vote” society, Google’s algorithm more closely models the shareholders of a corporation voting their shares; the votes of those with “more” (PageRank or shares) have a greater influence on the outcome. So a link from a page with a PageRank of 7 is more valuable than a link from a page of PageRank 3; probably many orders of magnitude more valuable, as you’ll see next.

The old 80/20 rule, on steroids

Because of the nature of the web, a small number of pages have a huge number of inbound links, and vice versa. So those with more links get more PageRank, but the value of PageRank is on a logarithmic scale thus it increases exponentially. Assuming[1] that the base were five (5), the value a page would get to vote based on it’s PageRank would look like this:

PageRank Value
0 0
1 5
2 25
3 125
4 625
5 3,125
6 15,625
7 78,125
8 390,625
9 1,953,125
10 9,765,625

An example:

Assume a site somehow manages to get a persistent link from MySpace’s home page (www.myspace.com). At the moment contains MySpace’s home page contains about 70 outbound links and has a PageRank of seven (7). Let’s also assume that there are a total of 50 other inbound links, and let’s say the average PageRank for those pages linking in is three (3) and those pages have an average outbound link count of 10. From this, let’s calculate PageRank:

MySpace’s Available PageRank per outbound link:
78,125 / 70 => 1,116
PageRank value contributed by 50 other sites:
125 * 50 / 10 => 625
Total PageRank value:
1,116 + 625 => 1,741

Looking it up in the table, the resultant PageRank for the home page is four (4).

The Three ‘P’s of Inbound Links

As with the three ‘L’s of real estate, the three ‘P’s of inbound links are: PageRank, PageRank, PageRank! [2] Note how in the prior example the 50 inbound links of PR3 offered less PageRank than the one (1) inbound link from MySpace with PR7! Of course we don’t know the logarithmic base [1] but Phil Craven says 5 or 6 are what many people believe it to be.

Here is what it would look like with base two (2) through ten (10) (download the full calculations here as a zipped Excel 2003 file [4kb]):

Logarithmic
Base
Value from
PR3 * 50 / 10
Value From
PR7 * 1 / 70
Total
Value
Resultant
PageRank
2 40 2 42 5
3 135 31 166 4
4 320 234 554 4
5 625 1,116 1741 4
6 1,080 3,999 5,079 4
7 1,715 11,765 13,480 4
8 2,560 29,959 32,519 4
9 3,645 68,328 71,973 5
10 5,000 142,857 147,867 5

So depending on the logarithmic base, PageRank fluctuates between four (4) and five (5) for this hypothetical example. However, starting with a logarithmic base of five (5) the one MySpace link overpowers the 50 others! And because pages with a PageRank closer to 10 are listed higher in Google’s search engine results page among competing pages, people focused on SEO are always trying to increase their page’s PageRank, often via unscrupulous means.

Of course nobody outside Google knows the exact formula or base exponents used, but hopefully this post illustrates the value of links from high PageRank pages.

Don’t game the system

However, I would be remiss if I didn’t point out that a single-minded focus on inbound links is fraught with peril, not the least because it might cause your pages to removed from Google’s index! Just as there are people selling weight loss products they claimn don’t require dieting or exercise, there are people offering ways to inbound links that don’t require having real people link to you. However, Google considers these shortcuts to be gaming the system is ever vigilant to discover those cheaters. If caught cheating, Google will ban your pages from their index without notice.

The best way to gain inbound links for your key pages on your website is to do the hard work of creating a site with great content that people want to link to.

Architecture Matters

So as an epilogue, getting inbound links is clearly necessary for high PageRank and thus good search engine results, but all those inbound links can be squandered without a good architecture and site management plan. To ensure that a site’s great content and popularity get reflected in appropriately high search engine ranking it’s critical to optimize the architecture of the site, the pages, and the URL structure as well as make plans for how the URL structure might change over time.

The most under-realized aspect of SEO

Personally, I think the most under-realized aspect of white hat SEO [3] is the lack of attention paid to URL planning and design, especially for larger websites. There are very few tools [4] besides the low-level and effectively simplistic URL rewriters like mod_rewrite for creating and maintaining a URL plan, very few articles [4] that discuss URL design, and no articles [4] I am aware of that discuss URL planning.

However, I believe website owners will see huge improvements compared to their prior rankings if they focus on URL design and create a URL management plan. The good news is that URL design is mostly a one time endeavor assuming site maintainers adhere to the management plan, at least until there is a full site rearchitecture.

But all of the whys and wherefores regarding URL planning and design are beyond the scope of this post, and instead will be the subject of many posts in the future. Stay subscribed!

For Further Research

And, as I stated at the start of this post you can learn more the PageRank formula here, and you can also google for PageRank to get a large list of other resources.

Footnotes

  1. But remember it’s a secret, so we can’t know for sure.
  2. There’s more to search engine ranking than just PageRank, like applicable content, but PageRank differentiates pages that compete competitive for the same keywords.
  3. To those SEO-haters of the world, please note that I’m referring to those things that you can do with pure white hat techniques, things that if not done can result in a great site being given less credit by the search engines.
  4. Over time, I plan to address the lack of such articles and tools for URL planning and design.

Technorati Tags: | | |

PayPal’s New API: So Close, Yet So Far

Friday, February 16th, 2007

I got an email from the PayPal Developer Network today announcing PayPal’s new “NVP” (or “Name-Value Pair“) API. Clearly they’ve learned that the complexity of SOAP is counter productive to adoption. Here’s what the email had to say about their new API:

NVP Is Your Integration MVP
We’re proud to announce that PayPal’s Name-Value Pair API has launched. Complex SOAP structure is now gone. All API methods are supported, except for AddressVerify. Get exclusive sample code – download two SDKs (Java and .NET). Get Details

Taking a look at their examples (in .ASP, .PHP, or ColdFusion) and their SDKs (for Java and ASP.NET [v1.1]) it’s nice to see they are using POST instead of GET. The following is one of their functions from their PHP examples (CallerService.php) that illustrates how their code is calling their NVP API (I edited for line-length only):

function hash_call($methodName,$nvpStr)
{
   //declaring of global variables
   global $API_Endpoint,$version,$API_UserName,
          $API_Password,$API_Signature,$nvp_Header;

   //setting the curl parameters.
   $ch = curl_init();
   curl_setopt($ch, CURLOPT_URL,$API_Endpoint);
   curl_setopt($ch, CURLOPT_VERBOSE, 1);

   //turning off the server and peer verification
   //(TrustManager Concept).
   curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
   curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
   curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
   curl_setopt($ch, CURLOPT_POST, 1);

   //if USE_PROXY constant set to TRUE in Constants.php,
   //then only proxy will be enabled. if USE_PROXY constant
   //set to TRUE in Constants.php, then only proxy will be
   //enabled.
   //Set proxy name to PROXY_HOST and port number to
   // PROXY_PORT in constants.php
   if(USE_PROXY)
      curl_setopt ($ch,
                        CURLOPT_PROXY,
                        PROXY_HOST.”:”.PROXY_PORT);

   //NVPRequest for submitting to server
   $nvpreq= “METHOD=”.urlencode($methodName).
            “&VERSION=”.urlencode($version).
            “&PWD=”.urlencode($API_Password).
            “&USER=”.urlencode($API_UserName).
            “&SIGNATURE=”.urlencode($API_Signature).$nvpStr;

   //setting the nvpreq as POST FIELD to curl
   curl_setopt($ch,CURLOPT_POSTFIELDS,$nvpreq);

   //getting response from server
   $response = curl_exec($ch);

   //convrting NVPResponse to an Associative Array
   $nvpResArray=deformatNVP($response);
   $nvpReqArray=deformatNVP($nvpreq);
   $_SESSION[’nvpReqArray’]=$nvpReqArray;

   if (curl_errno($ch)) {
        // moving to display page to display curl errors
        $_SESSION[’curl_error_no’]=curl_errno($ch) ;
        $_SESSION[’curl_error_msg’]=curl_error($ch);
        $location = “APIError.php”;
        header(”Location: $location”);
    } else {
        //closing the curl
        curl_close($ch);
     }

return $nvpResArray;
}

Much nicer and simplier than having to go to all the effort to set up a SOAP call.

Unfortunately, PayPal missed a huge opportunity to make their new API fully RESTful. Instead of designing a URL-centric REST interface (with a hypermedia component to keep the purist or the pure RESTafarians happy) they instead tunneled method calls over HTTP!  They used methods like “DoDirectPayment” and “RefundTransaction” Sheesh! (Note: these links to their methods load slower than any website I can remember visiting in ages and while loading my browser does lots of clicking. What the heck is going on in there I have no idea! You can go to the main docs via a much faster downloading PDF here. Wow, if that isn’t usually an oxymoron!)

Though much easier than calling SOAP, clearly not very RESTful.  Three steps forward, and two steps back. Sigh…

Bitten by the URI Opacity Axiom

Thursday, January 25th, 2007

 

Jon Udel has a post today entitled Divergent citation-indexing paths. Funny that he wrote about this; it seems he and I are on such a parallel trajectory these days. For evidence, take a look at my post from last week I titled Lessons Learned from Delicious Praise.

In his post Jon states “Del.icio.us, unlike Bloglines, treats the URLs that you feed to its citation counter in a case-sensitive way.” I wonder if Jon is aware of the Tim Berners-Lee’s URI Opacity Axiom?  A correct reading of the URI Opacity Axiom would reveal that in that Bloglines is in the wrong and Del.icio.us is in the right in this case; i.e. URLs are case-sensitive and programs SHOULD NOT[1] infer that two URLs with different casing point to the same resource. (NOTE: case doesn’t matter with domain names but it DOES matter in URL paths.)

As a matter of fact the URI Opacity Axiom is such as a closely held belief among those I like to call “the Weborati” that if you even question it so as to understand it on certain W3C or related mailing lists you’ll be in for a firestorm as if you blasphemed the messiah! ;-0

All kidding aside, especially since some of those people who hold the URI Opacity Axiom dear read this blog (!), after spending the time to research it and really learn it I came to believe that it is a very good idea for people to follow the URI Opacity Axiom. And I’ll discuss why in the future when I have more time. Unfortunately, like many principled concepts some people have elevated the URI Opacity Axiom to the level of dogma, and many of those who preach it believe it means a distortion of what it really means.

So Jon identified a real-world problem that following the URI Opacity Axiom introduces yet it is somewhat of a “catch-22“; following the axiom creates real-world problems but not following it creates other real world problems. But longer term, I really don’t think it has to be this way, and I’m working on ideas to address this issue that may turn into draft proposals or recommendations or something else. Basically I think with some “layers” of technology added to the web we could have the best of both worlds.

As an aside, Del.icio.us could update links based upon 301 redirects and then website owners could 301 redirect to a(n ideally lowercased) canonical URL whenever their server receives a request for a URL that is not in the canonical URLs format. This assumes of course that the website owner/server operator has chosen for their URLs to effectively be case-insensitive[2].

  1. I use the “SHOULD NOT” in the same way RFC 2119 defines the use of the uppercased term.  
  2. In my opinion running a website with case-sensitive URLs either means the web developer just wasn’t thinking or that they don’t have a clue about the affect case-sensitive URLs have on website usability. Ah, but that’s another subject for another day. :)

Proposing URI Templates for WebForms 2.0

Thursday, January 11th, 2007

I recently had an off-list email conversation with Ian Hickson, the editor of the Web Application Hypertext Technology Working Group specifications (i.e. HTML5 and WebForms 2.0). I was proposing to him that the current WebForms 2.0 be draft specification be amended to include a URI Template in the “action” attribute of the FORM element. Because I believe so strongly in the benefit of this proposal and because such things are inline with the Well Designed URLs Initiatitve was envisioned to advocate for, I decided to publish it to our blog and reference it in the WHATWG blog. The following is what I sent to Ian in email:

I really want to see WHATWG incorporate URI Templates for Web Form actions[1]. i.e.:

<form
action="http://foo.com/{make}/{model}/”
method="get">
<input type="text" name="make" />
<input type="text" name="mode" />
<input type="submit" />
</form>

If I type “Honda” and “Civic”, it will do a get to:

http://foo.com/Honda/Civic/

Instead of the only current possibility being something like:

<form
method="get"
action="http://foo.com/cars.php">
<input type="text" name="make" />
<input type="text" name="mode" />
<input type="submit" />
</form>

Which would produce the following for “Honda” and “Civic”:

http://foo.com/cars.php?make=Honda&model=Civic

To which Ian replied in two parts. Here is the first part:

“Why not just write a server-side redirector? That’s a trivial one to write. Four lines of code, maybe 10 if you make the recommended security checks first. You could also do it with a little bit of JavaScript.”

Unfortunately, a server-side redirector is not an appropriate solution in one case for the use-cases this proposal would address and doesn’t work for two others:

  1. Server-side redirection requires two round trips instead of one. I don’t believe any reasonably competent web architect for a high traffic site would allow a redirect for a high-traffic site. Consider any search engine such as the flagship offering for Ian’s employer; is Google likely to implement a redirect on every search request? Not likely. Server-side redirection reduces response time (by half?) and increases (doubles?) the number of concurrent requests that servers must handle. Using server-side redirection would probably also increase bandwidth requirements a measurable amount.
  2. It is not possible via HTTP to redirect the body of a POST. Consequently, the following use-case cannot be duplicated with a server-side redirect:

    <form
    action="http://www.myblog.com/{topic}/”
    method=”post”>
    <select name=”topic”>
    <option value=”first”>My 1st Post</option>
    <option value=”second”>My 2nd Post</option>
    <option value=”third”>My 3d Post</option>
    </select>
    <input type=”text” name=”comment”>
    <input type=”submit”>
    </form>

  3. For those wanting to create a form to direct to a website they do not control, a server-side redirect is absolutely not an option. For example, assume that I wanted to run a page on my website that lets people navigate to the topics on the WHATWG blog using a FORM with a SELECT? It’s simply not possible as WebForms 2.0 is currently specified without Javascript (addressed in a moment), but would be easily possible using a template (note I left off the closing “</option>” tags for formatting reasons):

    <form
    action="http://blog.whatwg.org/{topic}"
    method="post">
    <select name="topic">
    <option value="feed-autodiscovery">
    Feed Autodiscovery
    </option>
    <option
    value="text-content-checking">
    textContent Checking
    </option>
    <option value="checker-bug-fixes">
    Bug Fixes
    </option>
    <option
    value="significant-inline-checking">
    Significant Inline Checking
    </option>
    <option value="charmod-norm-checking">
    Charmod Norm Checking
    </option>
    <option
    value="proposing-features">
    Proposing features
    </option>
    </select>
    <input type="submit">
    </form>

  4. My belief is having this capability would encourage a lot more linking of this type between pages on the web.

So yes, server-side redirection is possible in some cases but by no means all, and for those cases where it’s possible it is not optimal.

Moving on the Ian’s suggestion to use “a little bit of JavaScript” to meet this use-case, I will admit it is possible to use JavaScript but these are the drawbacks in viewing JavaScript as the solution for this use-case:

  1. JavaScript is simply not allowed in a wide variety of web applications such as wikis, forums, and other sites that solicit community content although many of these do allow HTML elements such as FORM.
  2. Some users disable scripting on certain sites, often by decree of their employers.
  3. Javascript’s cross-browser compatibility issues make it less reliable and people are far less likely to depend on it when money is at stake, and forms are frequently used in those contexts.
  4. “User agents with no scripting support” from Section 1.3 Conformance Requirements of the Web Applications 1.0 Working Draft (HTML5) that incorporates WebForms 2.0. Need I say more?
  5. Declarative code is far easier to debug than procedural code.
  6. Far more people know HTML than JavasScript given the much greater skill required to master the latter, and that is unlikely to change.

It’s interesting to note that in the preface to the introduction for Section 3 of the WebForms 2.0 Working Draft of 12 October 2006, the following note is made about how everything that repeating form controls offers can already be done in JavaScript and the DOM. The mere fact that they went to the trouble to include something as complex as repeating controls into HTML5 when it can be done with JavaScript and the DOM implies that well-known patterns in web architecture are better implemented declaratively instead of via JavaScript and the DOM:

Occasionally forms contain repeating sections. For example, an order form could have one row per item, with product, quantity, and subtotal controls. The repeating form controls model defines how such a form can be described without resorting to scripting.

Note: The entire model can be emulated purely using JavaScript and the DOM. With such a library, this model could be used and down-level clients could be supported before user agents implemented it ubiquitously. Creating such a library is left as an exercise for the reader.

So yes it is possible to use JavaScript in many cases, but it no where near optimal. Javascript should not be considered the solution for as well-defined and obvious patterns such as submitting to a clean URL.

To further drive home the value of this proposal, anyone monitoring the REST-discuss list for any length of time will see that most REST experts tend toward using (what I call :) well-designed URLs, i.e. URLs where the resource is identified by path instead of query string. With WebForm 2.0’s pending support of PUT and DELETE, it would be just short of a crime not to include support for posting to clean URLs in WebForms 2.0.

Since having this discussion with Ian via email it was since pointed out to me on rest-discuss by Mark Baker that my proposal as written would break the existing web so was a non-starter. For some reason I wasn’t thinking about that, probably because I was more concerned about getting Ian (who I like to call: Mr. “No :) to agree that URI Templates were needed. Still, the solution would be simple.

What follows are my examples from above recast using an optional template attribute that would override the action attribute for WebForms 2.0 compliant browsers. This would of course require the server to accept both query string parameters and clean URLs (and hopefully do a server redirect from the former to the latter), or the submit could be implemented using Javascript for older browsers when applicable. Note that I didn’t show an example using JavaScript but, as the WebForm 2.0 spec says(that) is left as an exercise for the reader”:

  1. <form
    action="http://foo.com/model"
    template=”http://foo.com/{make}/{model}/”
    method=”get”>
    <input type=”text” name=”make” />
    <input type=”text” name=”mode” />
    <input type=”submit” />
    </form>

  2. <form
    action="http://www.myblog.com/topic"
    template=”http://www.myblog.com/{topic}/”
    method=”post”>
    <select name=”topic”>
    <option value=”first”>My 1st Post</option>
    <option value=”second”>My 2nd Post</option>
    <option value=”third”>My 3d Post</option>
    </select>
    <input type=”text” name=”comment”>
    <input type=”submit”>
    </form>

  3. <form
    action="http://blog.whatwg.org/topic"
    template=”http://blog.whatwg.org/{topic}”
    method="post">
    <select name="topic">
    <option value="feed-autodiscovery">
    Feed Autodiscovery
    </option>
    <option
    value="text-content-checking">
    textContent Checking
    </option>
    <option value="checker-bug-fixes">
    Bug Fixes
    </option>
    <option
    value="significant-inline-checking">
    Significant Inline Checking
    </option>
    <option value="charmod-norm-checking">
    Charmod Norm Checking
    </option>
    <option
    value="proposing-features">
    Proposing features
    </option>
    </select>
    <input type="submit">
    </form>

So in summary I really hope that Ian, who definitely seems to be the gatekeeper for what goes into HTML5 and what doesn’t go into HTML5, can see his way clear to add this feature to WebForms 2.0. If his main issue with it is needing to have it written up for inclusion in the spec, I’m more than happy to help.

About URI Templates

Wednesday, January 3rd, 2007

Probably one of the most interesting projects related to URL Design on which anyone is currently working is the URI Templates project spearheaded by Joe Gregorio of IBM. While not sexy and not something most end users will ever see, infrastructure developers writing blogs, wikis, forums, content management systems, and ideally even web servers, proxies, and routers will be able to utilize URI Templates for configuration and more. Well, once URI Templates are finalized and made an IETF standard that is.

Examples uses could include URL Rewriting Rules, URL Virtualization Configuration, REST’s Hypermedia Requirement, within future HTTP Headers and HTML LINK Elements, and even for Sitemaps (in a hopeful revision to Sitemap.org’s Sitemap protocol.) URI Templates will provide Internet professionals far more flexibility in dealing with URLs such as making it possible to employ URL Construction in a controlled manner without violating that nastly little URI Opacity Axiom.

Unlike most URL Rewriting Tools, URI Templates are very easy to use and, with the possible exception of certain esoteric edge cases that are still being finalized, accessible to mere mortals. URI Templates are certainly nowhere near as difficult as the requirement to master regular expressions and their complex interations when using Apache’s mod_rewrite and IIS’ (3rd Party) ISAPI Rewrite. Almost certainly anyone who has managed to language their own website will have no trouble at all mastering the most common use-cases for URI Templates.

URI Templates use a simple syntax where braces denote variables to be replaced when the templates are converted to actual URLs. For example, if this blog were using URI Templates to configure this page, the URI Template would most certainly look like this:

http://blog.welldesignedurls.org/{year}/{month}/{day}/{post-slug}/

When the template variables have the following values (as they do for this post):

year: 2007
month: 01
day: 03
post-slug: introducing-uri-templates

The resultant URL upon conversion will be:

http://blog.welldesignedurls.org/2007/01/03/introducing-uri-templates/

Pretty simple, huh?

However, be aware that before URI Templates can become an IETF standard there needs to be at least two interoperable implementations, if not a lot more. Mark Nottingham of Yahoo has implemented a URI Template parser in Javascript, but I don’t know for certain if that one will qualify for the purposes of standardization. Whatever the case, if you are working on any projects or products that could benefit from URI Templates I encourage you to participate in the URI working group via their mailing list and then implement the specification in your software projects or products.

Although it may appear to be a boring subject from the outside looking in, the standardization of URI Templates will be probably the most positive thing to happen for URL Design in over a decade.