Flo Ledermann

Towards the perfect URL scheme for web applications. Part 1: Requirements

2007-12-10T16:35:00.000+01:00

[This is part one of a four-part article. I will add links to the other parts here as they become available.]

I am currently working on a prototype for a small, new, top-secret web application (don't worry, I will let you know as soon as there is something online to show). When I sat down to start coding, the first thing I had to decide on was what the URLs inside the application should look like and how the URL space should be structured. I always had the impression that the question of how to design URLs for web applications has been largely neglected by the web developer community as I never read anything about a structured approach towards the problem — although it is a very visible feature and often the first usability test a site has to pass. So many apps and websites out there come with unreadable, unhackable URLs and/or, even worse, do not produce meaningful filenames when you try "save as..." on a page or downloadable file. I wanted to really get this right in the first place, so I started with writing down the things that annoy me regarding how URLs are constructed elsewhere, added a few "nice to have" features and lessons learned from my own previous experiences and compiled a requirements list that my scheme⁽¹⁾ should meet.

Requirements for a perfect URL scheme

When I talk about "modern web applications" here I have a rough concept of the state of the art in mind (buzzword alert!): AJAX interface with a fallback to server-driven interactivity, REST APIs⁽²⁾, widgetizable applications and an object-oriented data model. Other websites may have different requirements, but most of what I say here will apply to a broad range of applications.

1. URLs should be human readable, rememberable and robust for voice communication (e.g. phone)

One of the most important requirements for URLs is that they should make sense to a human. Machine-generated identifiers, deep folder hierarchies, special characters, strange extensions or query parameters are examples for URL parts that humans cannot easily parse or remember. More often than you think URLs are communicated by voice communication, over the phone or in the radio, therefore they should unambiguously serialze and deserialize from spoken language (which, by the way, is also a strong case for non-case sensitive or all-lowercase URLs).

Obviously this requirement cannot be solved by a specification or scheme alone, but also lies in the application's responsibility. For example, if you have a document with the name 'qa4@@67g$O', the best scheme will not help you to provide a human readable URL.

2. URLs should be short

While this requirement is slightly related to the previous one, it is not the same. A longer URL may be much easier to remember, as you can easily try by using one of these tinyurl.com generated URLs. Still, partly for the reason of transmitting and displaying on devices with limited capabilities (think text messages), the URLs of our system should be naturally short with no superfluous namespaces, extensions or query parameters.

3. The scheme should produce meaningful filenames & extensions for every page and file served

This is one of my top annoyances on the web, as most web applications fail to fulfill this requirement. Even with document servers that are supposed to provide files for download (e.g. PDFs), I frequently encounter missing extensions, not to speak of meaningful and unambiguous filenames. Whenever we serve a non-temporary page or document for viewing
it should be possible for the user to save that page to her local machine, getting a meaningful filename with the correct filename extension.

4. The scheme should make use of established specifications and conventions — filename extensions ('.'), hierarchy ('/'), queries ('?'), fragments ('#')

Obviously I do not want to reinvent URLs, but build on top of established standards and conventions. The problem is that these standards have emerged in a time before XML, Javascript and AJAX, and so we have to find an interpretation of the concepts offered to us that match the requirements of modern web applications.

5. The scheme should use as few reserved characters as possible, leaving most of the allowed character set for the application

I want to avoid introducing reserved characters in addition to the ones mentioned above - partly because it would be a nonstandard extension to what we already have, partly because human readability would suffer by introducing additional non-letter characters.

6. URLs should be "hackable"

Closely related to human parseability requested in requirement 1, "hackability" means that parts of the URL could be removed or added by users, leading to deterministic, meaningful results. Some examples of this would be: removing everything after the last '/' should give you an overview of the content one level up a hierarchy, removing or changing extensions should change the content type served, adding parts of an URL seen elsewhere on the site should result in similar behavior on any other resource etc.. Note that a system with "hackable" URLs should also protect the user from accidentally modifying or deleting resources this way, so that users can safely play around with the URL scheme without causing damage.

7. URLs should be able to address resources AND actions to be performed on these resources

This is the major new requirement I see with modern web applications. When HTTP was designed, URLs were meant as a way to access resources, in most cases static files. The operations to be performed on these resources were pushed into the HTTP protocol in form of the GET, PUT, POST, ... methods. In modern web applications it turned out that a predefined set of methods cannot cover all the different requirements that may emerge. We therefore need a way to express actions to be performed on resources within the URL, since no other method is provided by the HTTP protocol.

8. Resources or results of an action should be presentable in different formats (HTML, XML, JSON, plain text, ...), providing correct file name extensions

As static pages play a less important role in modern web applications and standards for the exchange of various data flavours emerge, it is important to be able to present a resource (e.g. an article) in different formats (e.g. HTML, Plain Text and RSS). New standards are emerging all the time and may become relevant in the future, so the pool of formats to choose from should be expandable in the future to support yet unanticipated usage of the data.

9. The scheme should support content to be served in a format determined by the preferences of the client application

In addition to the explicit choice of format through the URL scheme discussed in the last item, the mechanism of implicit content format selection offered by HTTP through the "Accept" header⁽³⁾ should be supported. The Accept header allows the client to send a list of supported content types along with each request, allowing the server software to select a content type best suited for the task.

This way our system can send different responses to a mobile phone and a full-fledged web browser, although both are using the same URL to access the content.

10. It should be possible to mix static files and generated content

Although static files play a less important role in modern web applications, it is obvious that every web application will need to serve static files form the file system to the client. While in production environments this is often solved by a dedicated subdomain (e.g. static.foo.com), it should still be possible to serve static files and generated content from a single server.

11. A single resource should have a single (normalized) URL

Flexible URL schemes often provide multiple paths to a single resource, leading to problems in unambiguously identifying resources. Search engines, for example, may not be able to conclude that two different URLs point to the same resource, usually leading to a reduced ranking for both pages. If multiple paths to a single resource are possible, an unambiguous normalization method (often called 'URL canonicalization') should be provided.

12. Support the features of "standard" global filenames like robots.txt, sitemap.xml, favicon.ico

Due to very unfortunate historical developments, some "standards" have emerged (none of them is actually a standard accepted by any official standardization authority) that rely on the presence of static files in the URL hierarchy of a server. Since these "standards" provide convenient or important functionality for many web applications, the URL scheme must either support these filenames by ensuring that no other resource can exist with an identical URL, or we need to find ways to support the features provided by these files through other means.

This concludes the requirements for a perfect URL scheme I have discovered and found important. In the next part (coming up soon!) I will discuss the constraints we face from the URL and HTTP specifications for designing our URLs. Part 3 will present the proposed URL scheme and Part 4 will discuss implementation issues. Stay tuned!

Footnotes

1 - Let me state in advance that when I say "URL scheme" here, I do not mean to propose a new scheme outside the scope of what HTTP already supports. My intention is to find a way to structure the path component (everything that comes after the server name and port in the URL) in a way that reflects the needs of modern web applications.

2 - I have to admit here I have never quite grasped what REST really is or why everyone is talking about it. As far as I understand, it is simply the concept of having a stateless API where every call is simply a HTTP request, and I have the vague idea that the URL scheme proposed here goes well along with it. If anyone has any further insigts, please enlighten me! (And yes, I have skimmed through the dissertation)

3 - The Accept header is discussed in detail in the HTTP/1.1 specification (RFC2616), section 14.

Updates

2007-12-19: Added numbering to the individual requirements and added requirement 7 (Hackability)

My talk at Web2Open this wednesday

2007-11-05T10:52:00.000+01:00

I will be giving a talk at Web2Open in Berlin (a free event attached to the O'Reilly Web2.0 Expo) this Wednesday. The announced title and abstract of the talk are:

Advanced GPolyline usage - performance issues and interactivity

Due to different implementations on different browsers, complex and high resolution Google polylines can lead to surprising results up to non-responsive applications in some cases. I will present the - sometimes surprising - results of our performance measurements, performance improvement strategies and a comparison with an experimental alternative implementation that can be also used for improved interactivity like real-time editing or continuous panning along a line.

But maybe you can best decide whether you are interested by looking at a screencast of the implementation I did for maptales. In the talk, I will provide some details and hints how to implement the shown features for your own applications.

Looking forward to see you at Web2Open!

Solving Hibernate Criteria's "Distinct Root Entity" limitation once and for all - including pagination

2007-10-28T12:34:00.001+01:00

I have to start with a rant: The Hibernate Criteria API, which I learned to at least partially like over the last months working on Maptales server code, has a few limitations which repeatedly make me wonder whether Hibernate really has been the right choice for a DB abstraction layer from time to time. It's not so much the limitations themselves, but rather the total lack of documentation of these limitations in the official Hibernate docs and examples and the generally rather arrogant tone on the Hibernate mailing Lists and FAQs — if you get an answer at all, that is — that annoy me and I guess a lot of other Hibernate users. The examples provided — and therefore copied blindly into nearly every Tutorial or Book about Hibernate out there — always avoid to show these limitations because of the triviality of the scenarios they cover. The purpose of a proper documentation would — for me — be to document especially such limitations and border cases where an API fails and provide solutions for these cases.

The problem is even amplified by the fact that googling for such limitiations rarely brings up a solution, since, as mentioned above, the hibernate documentation and inherited tutorials all do a good job in concealing the problems, and few coders out there seem to really use Hibernate in more sophisticated ways and publish their experiences. Which makes me wonder, because for these trivial cases too often good ol' SQL would have been perfectly sufficient — and advantageous to those lamer wannabee DB coders who would actually be forced to learn something about databases instead of replicating stupid Hibernate tutorial examples.

</rant>

OK now the specific limitation I want to talk about is: Hibernate does not return distinct results for a query with outer join fetching enabled for a collection. At least that's what they call it in the Hibernate advanced problems FAQ, failing to discuss this problem in deep and to give a general solution. Quote:

One day Hibernate might be smart enough to know that if you call setFirstResult() or setMaxResults() it should not use a join, but a second SQL SELECT. Try it, your version of Hibernate might already be smart enough. If not, write two queries, one for limiting stuff, the other for eager fetching.

And here they are again cutting of a large part of the problem. First, I checked an my version of hibernate is not "smart" enough, and it does not look like it will get any smarter soon. Second, the approach proposed there did not work for me when querying on properties of objects inside a collection - the collection has to always be JOINed for the query to work, therefore always producing results useless for pagination after transforming to distinct root entities. Generally, my problem was harder in a number of ways than the ususal examples:

I am nearly always using criterias on collection members, therefore I cannot revert to SELECT fetching.
I always have to use pagination (setFirstResult() and setMaxResults()), because in a web application it is not nice to return thousands of results on one page.
I am using PostGreSQL which adds a few oddities above it all (see below)

After digging through a lot of useless to semi-enlightening blog posts (thanks to Rick Hightower for providing two thirds of the solution) I could finally come up with the ultimate solution that works for me now:

First you have to accept that you have to live with 2 queries instead of one. You simply cannot do pagination and restrictions on collection members in hibernate in a single criteria query. Once you have accepted that fact, you can go ahead and construct your base query.

public List queryCats(Date queryDate) throws Exception {
Criteria criteria = getSession().createCriteria(Cat.class);
criteria.setFirstResult(offset).setMaxResults(num);
criteria.addOrder(Order.asc("age"));
if (queryDate != null) {
  Criteria subCrit = criteria.createCriteria("kittens", "kitten");
  subCrit.add(Expression.ge("birthDate", queryDate));
}

What you need to do then is to project the result to a collection of ids of the objects that match the query, eliminating duplicates. Here also the mentioned speciality of Postgres (at a second look it seems to be a general limitation) comes in: you have to include fields that are used for ordering in the projection clause. The nice thing is we can do the projection on demand and only if collection memebers are actually queried — this comes in handy for programatically constructed queries which can use the simple, single query way if collection properties are not used in the query:

  if (queryDate != null) {
  // project result to distinct ids,
  // including columns used for sorting
  criteria.setProjection(Projections.distinct(
    Projections.projectionList()
      .add(Projections.id())
      .add(Projections.property("birthDate"))
  ));
  List list = criteria.list();

  // unfortunately we have to copy the ids out of the
  // resulting Object[] List
  List idlist = new ArrayList<long>();

  for (Iterator iditer = list.iterator(); iditer.hasNext();) {
    Object[] record = (Object[]) iditer.next();
    idlist.add((Long)record[0]);
  }

  // another Hibernate stupidity: empty Lists cause
  // Expression.in to throw an error
  if (idlist.size() > 0) {
    criteria = getSession().createCriteria(Cat.class);
    criteria.add(Expression.in("id", idlist));
  }
  else {
    return new ArrayList();
  }
}

You can then transparently use the id-query in the complex case or issue the main query in the simple case:

  return criteria.list();
}

The presented approach has worked for me in a number of cases, and I am very happy now!