The thing to get out of the way upfront is that the easiest thing to extract look the ugliest and is often hard to follow. Its worth taking a few precautions to avoid the horror of machine generated RDF/XML.  So, install the Tabulator Firefox extension from MIT and find the button labelled “N3″ – it looks like a dense network icon. Hit that for a compact text based view, and un-toggle the default Tabular as screen space requires. The default is the loose network icon.

To actually extract the data, use the RDFa Distiller service. Put in a URL and this service gives you ugly RDF/XML by default, but the Tabulator extension comes to the rescue. With Tabulator hitting “Go” gives you – unsurprisingly – a table and switching completely to N3 is just two clicks.

In table mode, Tabulator will pick up and cache labels for things as it goes along and will use the last bit of the URL if it doesn’t have a label yet. If your URLs look ugly, then the view in Tabulator will look ugly – hopefully your URLs are pretty.

Pretty can also be bad, especially if the unique part of a URL is at the front. For example:

• http://feelitlive.com/events/2009/7/3/W2/2UH/Hyde+Park/Blur#event
• http://feelitlive.com/events/2009/7/4/HA9/0WS/Wembley+Stadium/Take+That#event
• http://feelitlive.com/events/2009/7/5/SW7/2AP/Royal+Albert+Hall/The+Killers#event

will all be rendered as “event” – very confusing! If this happens, switching to N3 may be the way forward. Part of the problem seems to be that Tabulator does not read RDFa on its own, which makes it harder to access the RDF and harder for Tabulator to calculate good labels. Apparently the next version will read RDFa – great.

Those links again:

• Tabulator Firefox extension
• RDFa Distiller service

The backward step was that some of the DataRSS content in the Yahoo Index appears to have vanished, this broke the examples in the previous post and on the Search Monkey site. I don’t think there is a lot a lowly web site operator like me can do to recover that, but I hope coverage will continue to be at least patchy.  I look forward to the day they just make the RDF available directly, I can see this being much more scalable for them and certainly easier for me.

At the same time I have made a little progress with adding hCalendar support. The level of adoption for hCalendar makes this compelling, though the lack of precision does mean its fundamentally restricted.  I will not, for example, be able to enhance results for pages that mention more than one event, even if one event is clearly (to a human) the primary topic of the page, choosing meaningful graphics is also circuitous.

On the topic of graphics, it is often not possible to depict an event that has not yet occurred. You might choose a graphic for purely aesthetic reasons, use an inconsistent rationale that is difficult to capture, or may abstract away the reasoning into another software module making it unavailable to the UI (as in my case).  This scenario seems to require a specific predicate which I will coin and document as I get around to it. A microformats equivalent is a non-starter as I do not have the inclination to make official representations to standards bodies for work of speculative value (see below).

In total then, the roadmap will be something like this (in order of priority):

1. Basic support for hCalendar (actually tested on upcoming), with simple safeguards for the ambiguous cases.
2. Support for RDFa enhanced pages mentioning multiple events, using foaf:primaryTopic to disambiguate.
3. Support for cases where the event itself can be previewed as a commercial proposition without requiring excess detail in the data, using the new predicate unless I can find one.
4. Support for event ontology actor and factor and associated foaf:depiction triples such that a factor or actor can be reliably depicted in the result.
5. Support for the organiser and logo hCalendar and hCard properties. This is useless to me as the interesting entity for music events is almost never the organiser and never the attendees [ unless you are on a date ], but its feasible so deserves a modicum of attention. I don’t think photo is worth supporting as that would immediately break on conferences.

I’m realistic about the importance of this project for making money as there is no predictable way to ensure large scale adoption of the plugin and the difference in click though rates is also unpredictable. As a result, I’ll be moving this forward primarily as a hackspace project rather than a commercial one, albeit commercially motivated. Doing it on hack evenings is a simple way to box off an amount of time.

Progress will be slow for other reasons, I have to locate implementations of the relevent vocabularies, trawl the Yahoo Index for fully crawled examples and possibly establish some test cases. This will all take time and much of it will be out of my control.

While apparently being in a position to know,  Mark Birkbeck “speculates” publicly that the UK Gov are using RDFa because:

by using RDFa to mark-up vacancies on each individual government site, it’s possible to allow each department to publish jobs however it sees fit. Many companies want to have some centralised information, not just government, but this usually involves imposing on each department some new database system or workflow. By using RDFa as the interface, each department merely needs to have the ability to publish HTML, and then they can share their data.

The second[reason] is that by publishing vacancies using RDFa, it’s easy for third-parties to ’scrape’ the data into their own databases, in a reliable way.

For example, some external company could import all vacancies for a particular region or of a particular type, and then show them on their own site.

Of course, all the benefits he mentioned a provably true – just run a page through a distiller – this is not bleeding edge technology any longer, it’s merely cutting edge.

One of the challenges with this work is that Yahoo has not given these widgets a particularly easy name to throw around, but here’s a picture to help explain things:

The text appears in a search result for Ultravox in London at the Roundhouse and as you can see it includes the usual snippet of text (or the event summary, if its longer) plus some very simple name value pairs – “Location” and “Starts”, which is altered to “Started” for events that have already started.

The idea is to answer the basic when and where questions common to events and also to allow users to quickly scan through search results and exclude events that they cannot attend and focus on those that they can attend, which is obviously those that they aren’t already missing! Therefore the “Starts” vs “Started” thing.

“Generic”

I can’t see a single person outside of the RDFa fan-club using this plugin unless it supports a wide variety of different sites and subject areas. There is no advantage to me in putting out something that only works for FeelItLive.com, yet its only FeelItLive.com that I’m aware of that is publishing VCal RDFa and Yahoo do not allow wildcards. I want people to use it, because I want them to focus their attention on my search results.

Chicken, meet Egg.

So, I’m doing two things:

• Labelling the thing generic – there is no point even mentioning FIL in the promotional blurb. It can only confuse matters.

• Inviting anyone and everyone to attach a comment to this post letting me know where they have published VCal RDFa. If they do that, I’ll do my best to support as many sites as Yahoo will allow.

I’ll also extend the same invitation to users of similar vocabularies (Edit: iCal, Event ontology, event microformats etc), though Vcal is the one recommended by Yahoo so I’ll have to see if anything else is supported.

Stripping her down to her triples we find out how we can add her to our network, find photos, that her and I are already in the same extended network  (well maybe in my dreams) and that she’s feeling productive right now (is that the same as being hot?).

More usefully, her friend count (203,412) and last login date (Tuesday) are both published in the only de jure machine readable meta-data format that we know – for sure – can publish just about any fact.