More Merging with SPARQL
Tuesday, September 12, 2006
Last week, I queried a scruffy little RDB with SPARQL, using SquirrelRDF.
This week, with a bit of help from the Jena team, I've been merging data across sources, where one is the shiny new triplestore, and one the scruffyLittleRDB.
I wanted to write a query to tell us where we've sent a particular article header, and when. (We push data off to Crossref, Ebsco, etc etc.)
To answer this question, I needed:
1. Literal data about articles from the big triplestore. I picked on article /333333. As you can see, it was published in volume 33, number 4, of the Journal "Behavior Research Methods, Instruments, & Computers".
2. Who-which-where data from the distribution database. It also has a notion of journals and issues - this issue is the same one as my 'article 333333' above belongs to. It also knows who we've sent that issue to, and when.
I extended my SPARQL Service to allow named models which are actually (known) SquirrelRDF sources.
Here's my query, (or a screenshot of it anyway):
(Scared yet?)
And here are the results.
I promised to report my findings as I went along, so this was my experience with this latest experiment ...
1. Merging is all very dependent on finding suitable identifiers on which to join.
If you look at the query , you'll see I'm identifying an article by the ISSN of the journal it is from, the journal volume, and the journal issue. I'm joining on all three to get some identity between the two sources. This makes for a complex query. But there are no other options, (or none I can think of!) The scruffyLittleRDB has its own identifier system.. the traditional autoincrementing field. The triplestore, no matter how clever, can never guess these identifiers - and so we are reduced to poking around with literals. Things get even nastier when you consider datatypes - note I'm translating with a FILTER.
2. SPARQL can get quite fierce.
SPARQL Queries can be a bit - well, big! Writing real-world SPARQL reminds me of writing regexes; I start itching to comment every line. I've scared two SQL engineers away from my screen with that one above already!
As ever, more to come..
This week, with a bit of help from the Jena team, I've been merging data across sources, where one is the shiny new triplestore, and one the scruffyLittleRDB.
I wanted to write a query to tell us where we've sent a particular article header, and when. (We push data off to Crossref, Ebsco, etc etc.)
To answer this question, I needed:
1. Literal data about articles from the big triplestore. I picked on article /333333. As you can see, it was published in volume 33, number 4, of the Journal "Behavior Research Methods, Instruments, & Computers".
2. Who-which-where data from the distribution database. It also has a notion of journals and issues - this issue is the same one as my 'article 333333' above belongs to. It also knows who we've sent that issue to, and when.
I extended my SPARQL Service to allow named models which are actually (known) SquirrelRDF sources.
Here's my query, (or a screenshot of it anyway):
(Scared yet?)
And here are the results.
I promised to report my findings as I went along, so this was my experience with this latest experiment ...
1. Merging is all very dependent on finding suitable identifiers on which to join.
If you look at the query , you'll see I'm identifying an article by the ISSN of the journal it is from, the journal volume, and the journal issue. I'm joining on all three to get some identity between the two sources. This makes for a complex query. But there are no other options, (or none I can think of!) The scruffyLittleRDB has its own identifier system.. the traditional autoincrementing field. The triplestore, no matter how clever, can never guess these identifiers - and so we are reduced to poking around with literals. Things get even nastier when you consider datatypes - note I'm translating with a FILTER.
2. SPARQL can get quite fierce.
SPARQL Queries can be a bit - well, big! Writing real-world SPARQL reminds me of writing regexes; I start itching to comment every line. I've scared two SQL engineers away from my screen with that one above already!
As ever, more to come..
posted by Katie Portwin at 3:40 pm
Amazon + Ingenta + SPARQL = nifty
Friday, April 21, 2006
This week, I've been playing with merging multiple RDF data sources, and doing SPARQL queries across the set. This should let us offer enhanced metadata on the website...
Here's the imaginary future IngentaConnect abstract page (in an audio browser):
What's that you say user? You enjoyed reading "Obesity and Poverty" on Ingenta? Did you know that people who bought it on Amazon, also bought... What's that? You're thinking of buying a hard copy so you can read it again in the bath? Well it's up to you, but now it's £28 from the big yellow place..."
"Hey, pal, I notice you're reading "Realising the Full Potential of the Web" on Ingenta Do you realise that's by Tim Berners-Lee? Did you know that he's geolocated at 42.361860,-71.091840? eh?
In order to experiment with merge queries, I first had a go at implementing the SPARQL Protocol using Jena.
You submit a GET which goes:
(line breaks added for clarity)
For default-graphs, the service tries to open a connection to a known triplestore, for named-graphs it requests the document and reads it.
Next, I set up some demos:
The demo is a form containing the query. Here's the the results (sorry it's a flat copy - can't open it up for real yet).
Data is merged from 3 sources:
1. Book from metastore XML API(Flat copy).
2. Book from Amazon (this is the real service live btw.)
I made this nice RDF/XML by URL pipelining standard stuff from their API with an XSLT stylesheet.
3. An owl:sameAs statement - to hook em together.
Here's the query, and the results.
Data is merged from 2 sources:
1. Metastore XML API: Article, Author
2. An RDF business card document on the web
The two are merged by joining on surname.
I'll open the API for experimentation as soon as I can get it past the legal people. Registration of interest in that, comments, and suggestions for other merges, very welcome.
Here's the imaginary future IngentaConnect abstract page (in an audio browser):
What's that you say user? You enjoyed reading "Obesity and Poverty" on Ingenta? Did you know that people who bought it on Amazon, also bought... What's that? You're thinking of buying a hard copy so you can read it again in the bath? Well it's up to you, but now it's £28 from the big yellow place..."
"Hey, pal, I notice you're reading "Realising the Full Potential of the Web" on Ingenta Do you realise that's by Tim Berners-Lee? Did you know that he's geolocated at 42.361860,-71.091840? eh?
In order to experiment with merge queries, I first had a go at implementing the SPARQL Protocol using Jena.
You submit a GET which goes:
query=[Scary SPARQL query]
& default-graph-uri=[metastore.ingenta.com]
& named-graph-uri=[URI of some other RDF data]
(line breaks added for clarity)
For default-graphs, the service tries to open a connection to a known triplestore, for named-graphs it requests the document and reads it.
Next, I set up some demos:
Example 1. Merging with Amazon to enhance Book data
The demo is a form containing the query. Here's the the results (sorry it's a flat copy - can't open it up for real yet).
Data is merged from 3 sources:
1. Book from metastore XML API(Flat copy).
2. Book from Amazon (this is the real service live btw.)
I made this nice RDF/XML by URL pipelining standard stuff from their API with an XSLT stylesheet.
3. An owl:sameAs statement - to hook em together.
2. Merging with personal FOAF documents to enhance Author data
Here's the query, and the results.
Data is merged from 2 sources:
1. Metastore XML API: Article, Author
2. An RDF business card document on the web
The two are merged by joining on surname.
I'll open the API for experimentation as soon as I can get it past the legal people. Registration of interest in that, comments, and suggestions for other merges, very welcome.
posted by Katie Portwin at 4:34 pm
How does a scruffy little RDB get into a Triplestores-Only Golf Club?
Thursday, September 07, 2006
This week, I've been playing with SquirrelRDF. It allows you to query RDBs with SPARQL. Magic.
Why? Well, there are lots of RDBs full of interesting metadata, lurking under rocks here at Ingenta. It's my job to worry about integrating metadata, and for us, the future is RDF.
I could tip ALL the data into my nice new triplestore. All your databases will be ASSSIMILATED. Resistance is futile, etc etc. And that's probably the way to go with most of it.
But in some cases, this approach just isn't appropriate. What if the owner guards it jealously and I'm waiting for them to go on leave before we snaffle it? What if the data is itself a bit freaky, and I just don't want it polluting my store? What if I haven't written the loader yet. (Writing loaders is a lot of work!) What if there's just butt-loads of it, so the load is going to take 6 weeks of processing? I want that data NOW!
I decided to experiment with SquirrelRDF (Although D2RQ would be another option.) It implements this spec, and works by treating each row as a resource, where the column names are its properties, and the column values are the objects.
Eg:
I have a database called POND, and in it, a table called FROGS which goes
_x ns:FROGS_ID 1
_x ns:FROGS_name "Fred"
_x ns:FROGS_colour "green"
_x ns:FROGS_legs 4
_y ns:FROGS_ID 2
...etc..
Where "ns" is some URI I configure - eg http://www.ingenta.com/POND/
One nice advantage is I don't have to define a new vocabulary for Frogs - that's all automatic, based on the db schema.
Overall, it was a good out-of-the box experience. I downloaded the jars, and ran a command line utility which generated this maping file. It seemed to handle the datatypes OK, and made the automatic vocabulary. Next, I was able to run a SPARQL query using another command line utility. Finally, I installed the little webapp, and did the same thing over HTTP; I entered this query in the box, and got these results.
Of course, it won't be able to offer the flexibility of a real RDF store - it will still be a faff to add a new property etc. But, at least I should be able to do SPARQL on it, and start joining it up with other data from the triplestore, or from under other rocks.
Why? Well, there are lots of RDBs full of interesting metadata, lurking under rocks here at Ingenta. It's my job to worry about integrating metadata, and for us, the future is RDF.
I could tip ALL the data into my nice new triplestore. All your databases will be ASSSIMILATED. Resistance is futile, etc etc. And that's probably the way to go with most of it.
But in some cases, this approach just isn't appropriate. What if the owner guards it jealously and I'm waiting for them to go on leave before we snaffle it? What if the data is itself a bit freaky, and I just don't want it polluting my store? What if I haven't written the loader yet. (Writing loaders is a lot of work!) What if there's just butt-loads of it, so the load is going to take 6 weeks of processing? I want that data NOW!
I decided to experiment with SquirrelRDF (Although D2RQ would be another option.) It implements this spec, and works by treating each row as a resource, where the column names are its properties, and the column values are the objects.
Eg:
I have a database called POND, and in it, a table called FROGS which goes
This gets translated to triples:
ID | name | colour | legs
----------------------------
1 | Fred | green | 4
2 | Felix | yellow | 4
3 | Frank | green | 3
_x ns:FROGS_ID 1
_x ns:FROGS_name "Fred"
_x ns:FROGS_colour "green"
_x ns:FROGS_legs 4
_y ns:FROGS_ID 2
...etc..
Where "ns" is some URI I configure - eg http://www.ingenta.com/POND/
One nice advantage is I don't have to define a new vocabulary for Frogs - that's all automatic, based on the db schema.
Overall, it was a good out-of-the box experience. I downloaded the jars, and ran a command line utility which generated this maping file. It seemed to handle the datatypes OK, and made the automatic vocabulary. Next, I was able to run a SPARQL query using another command line utility. Finally, I installed the little webapp, and did the same thing over HTTP; I entered this query in the box, and got these results.
Of course, it won't be able to offer the flexibility of a real RDF store - it will still be a faff to add a new property etc. But, at least I should be able to do SPARQL on it, and start joining it up with other data from the triplestore, or from under other rocks.
posted by Katie Portwin at 10:00 am
performance, triplestores, and going round in circles..
Wednesday, May 24, 2006
Can industrial scale triplestores be made to perform?
Is breaking the "triple table" model the answer?
Scaling SPARQL performance is something I worry about. Mike Bergman worries about it too.
In our XTech paper, we showed that even a simple, bread and butter sample query, is taking 1.5 seconds on our 200million triplestore. (Scroll down for the colourful graph). In our Jena paper, we showed that with a triplestore in a relational database, you're at the mercy of your query planner - some configurations of SPARQL query perform reaaaaly badly.
Both these facts make sense when you think that your SPARQL doesn't just get done by magic - of course, it gets turned into a ma-hoosive SQL JOIN statement, across the main triples table again and again, like this:
Databases weren't really designed to do queries like this - it's not surprising that they aren't very fast at it...
So I was really interested by Kevin Wilkinson's paper on property tables - also at the JUC last week. Kevin's idea is basically this:
Say, in your triplestore, you have this:
(This means that X has three properties attached to it. )
Kevin points out that often, you get "groups" - patterns in the data. Often, X's always have exactly one o1, one o2 and one o3.
For example, consider X to be an article, and articles always have one title, one doi, one volume, issue, etc.
In this case, you could have a "property" table like this:
You could still keep other unsuitable-for-grouping stuff in the main triples table, but drag out groups into 'property tables'. This would make them fast to query. (There's a lot more to it than this - eg how properties with cardinality >1 go in their own table too - but you'll have to read the paper!)
This made sense to me. There are definitely patterns in the data. We identify articles by their issn/volume/issue/page data. They always(?) have these properties. How sensible it would seem to put them in a nice normal table, so we can at least occasionally have sql like:
- rather than scary mega-joins, to throw at our poor old database. Sounds totally common sense!
BUT
....
....
Hangon....
Consider this: our 'articles' table would look like this:
Er.... does this look rather familiar? Isn't this exactly what we started with?
Hang on - what was wrong with what we started with again?
Well, for us, the problem was that we found that we had to keep re-modelling the database - initial assumptions were correct for the first hundred thousand articles.. and then suddenly weren't. For example, we recently discovered that sometimes Articles have 2 DOIs. Horrible, but true. The great thing about the triplestore is that we don't have to bake assumptions about the data into the database - we can have as many whatevers as we like.
I'm still very keen to try out Kevin's architecture - I hope to nag him into solving this undecided-cardinality-problem within his app by changing the schema dynamically and shufting the data around. I just hope that could scale. Much looking forward to first release and finding out more!
Is breaking the "triple table" model the answer?
Scaling SPARQL performance is something I worry about. Mike Bergman worries about it too.
In our XTech paper, we showed that even a simple, bread and butter sample query, is taking 1.5 seconds on our 200million triplestore. (Scroll down for the colourful graph). In our Jena paper, we showed that with a triplestore in a relational database, you're at the mercy of your query planner - some configurations of SPARQL query perform reaaaaly badly.
Both these facts make sense when you think that your SPARQL doesn't just get done by magic - of course, it gets turned into a ma-hoosive SQL JOIN statement, across the main triples table again and again, like this:
Select A0.Subj, A2.Subj, A3.Obj, A4.Obj, A5.Subj, A6.Obj From jena_g1t1_stmt A0, jena_g1t1_stmt A1,
jena_g1t1_stmt A2, jena_g1t1_stmt A3, jena_g1t1_stmt A4, jena_g1t1_stmt A5, jena_g1t1_stmt A6 Where
A0.Prop=...Databases weren't really designed to do queries like this - it's not surprising that they aren't very fast at it...
So I was really interested by Kevin Wilkinson's paper on property tables - also at the JUC last week. Kevin's idea is basically this:
Say, in your triplestore, you have this:
triples table
---------------
| s | p | o |
---------------
| X | p1 | o1 |
| X | p2 | o2 |
| X | p3 | o3 |
| Y | p1y| o1y|
(This means that X has three properties attached to it. )
Kevin points out that often, you get "groups" - patterns in the data. Often, X's always have exactly one o1, one o2 and one o3.
For example, consider X to be an article, and articles always have one title, one doi, one volume, issue, etc.
In this case, you could have a "property" table like this:
------------------
s | p1 | p2 | p3 |
------------------
X | o1 | o2 | o3 |
Y | o1y| o2y| o3y|
You could still keep other unsuitable-for-grouping stuff in the main triples table, but drag out groups into 'property tables'. This would make them fast to query. (There's a lot more to it than this - eg how properties with cardinality >1 go in their own table too - but you'll have to read the paper!)
This made sense to me. There are definitely patterns in the data. We identify articles by their issn/volume/issue/page data. They always(?) have these properties. How sensible it would seem to put them in a nice normal table, so we can at least occasionally have sql like:
"select * from articles where articles.vol=...." - rather than scary mega-joins, to throw at our poor old database. Sounds totally common sense!
BUT
....
....
Hangon....
Consider this: our 'articles' table would look like this:
Articles
---------------------------------------
id | title | doi | volume | .....
---------------------------------------
1 | "bla" | 123.456..| 42 | .......
2 | "foo" | 789.101..| 45 | .......
Er.... does this look rather familiar? Isn't this exactly what we started with?
Hang on - what was wrong with what we started with again?
Well, for us, the problem was that we found that we had to keep re-modelling the database - initial assumptions were correct for the first hundred thousand articles.. and then suddenly weren't. For example, we recently discovered that sometimes Articles have 2 DOIs. Horrible, but true. The great thing about the triplestore is that we don't have to bake assumptions about the data into the database - we can have as many whatevers as we like.
I'm still very keen to try out Kevin's architecture - I hope to nag him into solving this undecided-cardinality-problem within his app by changing the schema dynamically and shufting the data around. I just hope that could scale. Much looking forward to first release and finding out more!
posted by Katie Portwin at 1:44 pm
Paper at XTech
Friday, February 24, 2006
The MetaStore project team have had a paper accepted at the XTech 2006 conference in Amsterdam.
XTech (formerly XML Europe) is all about XML technologies; topics include Semantic Web and RDF, Tagging, Annotation, Mashups, Web Services - all the new fangled "Web 2.0" stuff...
It is apparently "the premier European conference for developers, information designers and managers working with web and standards-based technologies." (yikes!) The keynote speakers are from Amazon and Yahoo. (double yikes!).
This is the abstract:
The aim of the Ingenta MetaStore project is to build a flexible and scalable repository for the storage of bibliographic metadata spanning 17 million articles and 20,000 publications.
The repository replaces several existing data stores and will act as a focal point for integration of a number of existing applications and future projects. Scalability, replication and robustness were important considerations in the repository design.
After introducing the benefits of using RDF as the data model for this repository, the paper will focus on the practical challenges involved in creating and managing a very large triple store.
The repository currently contains over 200 million triples from a range of vocabularies including FOAF, Dublin Core and PRISM.
The challenges faced range from schema design, data loading, SPARQL query performance. Load testing of the repository provided some insights into the tuning of SPARQL queries.
The paper will introduce the solutions developed to meet these challenges with the goal of helping others seeking to deploy a large triple store in a production environment. The paper will also suggest some avenues for further research and development.
Now we just need to write and deliver a presentation... eep! In case it all goes horribly wrong, I'm now recruiting for a volunteer member of the audience to faint at my signal...
XTech (formerly XML Europe) is all about XML technologies; topics include Semantic Web and RDF, Tagging, Annotation, Mashups, Web Services - all the new fangled "Web 2.0" stuff...
It is apparently "the premier European conference for developers, information designers and managers working with web and standards-based technologies." (yikes!) The keynote speakers are from Amazon and Yahoo. (double yikes!).
This is the abstract:
The aim of the Ingenta MetaStore project is to build a flexible and scalable repository for the storage of bibliographic metadata spanning 17 million articles and 20,000 publications.
The repository replaces several existing data stores and will act as a focal point for integration of a number of existing applications and future projects. Scalability, replication and robustness were important considerations in the repository design.
After introducing the benefits of using RDF as the data model for this repository, the paper will focus on the practical challenges involved in creating and managing a very large triple store.
The repository currently contains over 200 million triples from a range of vocabularies including FOAF, Dublin Core and PRISM.
The challenges faced range from schema design, data loading, SPARQL query performance. Load testing of the repository provided some insights into the tuning of SPARQL queries.
The paper will introduce the solutions developed to meet these challenges with the goal of helping others seeking to deploy a large triple store in a production environment. The paper will also suggest some avenues for further research and development.
Now we just need to write and deliver a presentation... eep! In case it all goes horribly wrong, I'm now recruiting for a volunteer member of the audience to faint at my signal...
posted by Katie Portwin at 3:44 pm
Connect Compilations - a little glimpse of the future
Monday, October 06, 2008
Connect compilations will be introduced into ingentaconnect during the next couple of weeks, its features represent a delicious sample of the technical offering Publishing Technology is cooking up.
Connect Compilations enable publishers to assemble 'virtual' publications from their existing content on connect. Compilations are given titles, descriptions, links and logos such that they look similar to conventional publications. They may be organised in familiar serial and monograph formats. At ingentaconnect Compilations may be purchased and subscribed to in the same way as other publications. Crucially the publisher has control over the Compilation, it is available to amend and augment whenever they please.
From an end user perspective Connect Compilations will be quietly integrated into the search and browse facilities on connect. For publishers the changes are more marked, a whole set of administration tools have been introduced.
To provide powerful administration tools we've increased our adoption of client side plugins (based on Jquery) and paradigms like AJAX. Both have been on the list of 'must have' technical buzz words for a some time, but we've taken care only to employ them where there is tangible benefit. Most significant is the introduction of semantic technologies; an RDF triple store for data, SPARQL to query it, Jena and our own framework to represent data to the application.
One may well ask what immediate benefit does semantic technology bring, beyond exciting programmers and web luminaries? The first benefit we'll see on ingentaconnect is tighter integration, both inside the site and with the wider web. RDF enables us to make assertions about resources (like articles, authors and references) without imposing constraints on the assertions made, or how they will be used. Crucially we can use the assertions to draw conclusions, or inferences, to fill in gaps, and really 'understand' the data. All of this is achieved with little redundancy or repetition. The factors combine to produce a store on which services to cater for varying requirements and perspectives can readily be built.
The benefits I've mentioned thus far could of course be realised with a relational database, but we're laying our new foundations at present, and more will grow out them.
In down to earth speak, all this means ingentaconnect, and close relation pub2web, will increasingly provide accurate linking, interesting ways to splice together content and, as Connect Compilations demonstrates, put control into the hands of online Publishers.
Connect Compilations enable publishers to assemble 'virtual' publications from their existing content on connect. Compilations are given titles, descriptions, links and logos such that they look similar to conventional publications. They may be organised in familiar serial and monograph formats. At ingentaconnect Compilations may be purchased and subscribed to in the same way as other publications. Crucially the publisher has control over the Compilation, it is available to amend and augment whenever they please.
From an end user perspective Connect Compilations will be quietly integrated into the search and browse facilities on connect. For publishers the changes are more marked, a whole set of administration tools have been introduced.
To provide powerful administration tools we've increased our adoption of client side plugins (based on Jquery) and paradigms like AJAX. Both have been on the list of 'must have' technical buzz words for a some time, but we've taken care only to employ them where there is tangible benefit. Most significant is the introduction of semantic technologies; an RDF triple store for data, SPARQL to query it, Jena and our own framework to represent data to the application.
One may well ask what immediate benefit does semantic technology bring, beyond exciting programmers and web luminaries? The first benefit we'll see on ingentaconnect is tighter integration, both inside the site and with the wider web. RDF enables us to make assertions about resources (like articles, authors and references) without imposing constraints on the assertions made, or how they will be used. Crucially we can use the assertions to draw conclusions, or inferences, to fill in gaps, and really 'understand' the data. All of this is achieved with little redundancy or repetition. The factors combine to produce a store on which services to cater for varying requirements and perspectives can readily be built.
The benefits I've mentioned thus far could of course be realised with a relational database, but we're laying our new foundations at present, and more will grow out them.
In down to earth speak, all this means ingentaconnect, and close relation pub2web, will increasingly provide accurate linking, interesting ways to splice together content and, as Connect Compilations demonstrates, put control into the hands of online Publishers.
Labels: connect compilations, ingentaconnect, pub2web, publishing, publishing technology, semantic, technology
posted by John Clapham at 8:55 am
TripleStore develops mental capacity of goldfish
Thursday, November 02, 2006
This week in the MetaStore team we've added inferencing using an OWL 1 Ontology.
An ontology is a statement of what there is, and how things fit together. Eg,
* In Ingenta-land, there are Articles, they have Keywords and Authors, they are published in Journals. Every Book must have at least one Author.
* In My-lunch-land, there are sandwiches, chocolate bars, cups of tea. Every sandwich must have exactly two and only two slices of bread.
Inferencing, in this context, means guessing new facts, based on known facts, using logical rules in an ontology.
So. If we have a database with the fact that John Steinbeck wrote Sweet Thursday, and an ontology which says that being an author is the inverse of having an author, then a computer can, all on its own, reason that Sweet Thursday was written by John Steinbeck. Super eh! Hal here we come.
Here's the machine-readable version:
At the beginning of this project, we were very excited about OWL. We planned to mine new information out of our scholarly research data set. For example, if Author Bob wrote article A, and article B, and Author Bill collaborated with Bob on C, and wrote D on his own, perhaps A and D are related. Or was that B and D? My brain hurts.. either way, you get the picture...
The problem with was, as usual with our project, scalability. The Jena inferencer choked at 11 million triples... 190 million away from our full load.
Last week, Priya and I came up with a practical solution to this problem: a two stage approach.
1. Guess what bit of the model you might be intersted in, and hold a bit of it in memory. (Techie detail: I implemented this using a SPARQL CONSTRUCT query like this, and stored them in a Jena model.)
2. Give that to the Jena Inferencer to chew on, instead of the big fat data set.
Obviously, the success of this approach depends on how good your guess is in 1.
So, last week, I knew that this article was written by J Bhatt; this week, I know that he wrote all these too. Last week, I knew that this article was about bananas. This week, I know that so are all these.
1. Web Ontology Language (There is some explanation to do with Winnie the Pooh for why it isn't 'WOL'.. basically, it just sounded foolish).
An ontology is a statement of what there is, and how things fit together. Eg,
* In Ingenta-land, there are Articles, they have Keywords and Authors, they are published in Journals. Every Book must have at least one Author.
* In My-lunch-land, there are sandwiches, chocolate bars, cups of tea. Every sandwich must have exactly two and only two slices of bread.
Inferencing, in this context, means guessing new facts, based on known facts, using logical rules in an ontology.
So. If we have a database with the fact that John Steinbeck wrote Sweet Thursday, and an ontology which says that being an author is the inverse of having an author, then a computer can, all on its own, reason that Sweet Thursday was written by John Steinbeck. Super eh! Hal here we come.
Here's the machine-readable version:
SweetThursday foaf:maker JohnSteinbeck .
+
<owl:ObjectProperty rdf:about="&foaf;made">
<owl:inverseOf rdf:resource="&foaf;maker"/>
</owl:ObjectProperty>
+
Jena Reasoner
=
JohnSteinbeck foaf:made SweetThursday .
At the beginning of this project, we were very excited about OWL. We planned to mine new information out of our scholarly research data set. For example, if Author Bob wrote article A, and article B, and Author Bill collaborated with Bob on C, and wrote D on his own, perhaps A and D are related. Or was that B and D? My brain hurts.. either way, you get the picture...
The problem with was, as usual with our project, scalability. The Jena inferencer choked at 11 million triples... 190 million away from our full load.
Last week, Priya and I came up with a practical solution to this problem: a two stage approach.
1. Guess what bit of the model you might be intersted in, and hold a bit of it in memory. (Techie detail: I implemented this using a SPARQL CONSTRUCT query like this, and stored them in a Jena model.)
2. Give that to the Jena Inferencer to chew on, instead of the big fat data set.
Obviously, the success of this approach depends on how good your guess is in 1.
So, last week, I knew that this article was written by J Bhatt; this week, I know that he wrote all these too. Last week, I knew that this article was about bananas. This week, I know that so are all these.
1. Web Ontology Language (There is some explanation to do with Winnie the Pooh for why it isn't 'WOL'.. basically, it just sounded foolish).
posted by Katie Portwin at 5:25 pm
Openacademia
Tuesday, June 13, 2006
Yesterday an announcement went out to the Semantic Web interest group mailing list introducing openacademia 1.0, "an RDF-based publication repository for research groups and scientific communities".
As the documentation explains, openacademia "is an open source publication metadata repository for scientific communities. Our goal is to allow scientists to collect, organize and disseminate publications more efficiently using a combination of novel semantic technologies". The software which can be run locally, e.g. within a group of institutions, allows users to deposit metadata about their publications which can then be shared with others as RSS feeds, etc.
The software supports BibText, which is used by many scholars to maintain and format bibliographies, making it easy to add data to the system. APIs are available to extract data from the repository (which is based on Sesame). There are some nice tools for visualising topics between papers, graphs of author relationships, etc. The RSS feeds contain detailed metadata about each paper and author.
I'm going to have a more detailed look at the system over the coming weeks, but it intrigues me as their approach echoes our own adoption of RDF and Semantic Web technologies for aggregating and storing RDF metadata. Indeed many of the vocabularies are also shared, for example they're using FOAF to describe authors. The more publishing data that becomes available as RDF, the better the network effects become.
We're currently considering ways to open up our own repository for people to explore. Part of the technology partner role we play for publishers is ensuring that their content reaches the widest audience and is accessible as possible. With a growing number of applications exposing Semantic Web data, access to RDF and SPARQL interfaces will be just as important as "traditional" and community specific interfaces such as OpenURL, and ZING. Historically there's been some disconnects between the library and publisher communities on how they store, manage and share metadata. As we're firmly in-between these communities I'm hoping we can start to help bridge these gaps, and I'm convinced that Semantic Web technologies can help.
As the documentation explains, openacademia "is an open source publication metadata repository for scientific communities. Our goal is to allow scientists to collect, organize and disseminate publications more efficiently using a combination of novel semantic technologies". The software which can be run locally, e.g. within a group of institutions, allows users to deposit metadata about their publications which can then be shared with others as RSS feeds, etc.
The software supports BibText, which is used by many scholars to maintain and format bibliographies, making it easy to add data to the system. APIs are available to extract data from the repository (which is based on Sesame). There are some nice tools for visualising topics between papers, graphs of author relationships, etc. The RSS feeds contain detailed metadata about each paper and author.
I'm going to have a more detailed look at the system over the coming weeks, but it intrigues me as their approach echoes our own adoption of RDF and Semantic Web technologies for aggregating and storing RDF metadata. Indeed many of the vocabularies are also shared, for example they're using FOAF to describe authors. The more publishing data that becomes available as RDF, the better the network effects become.
We're currently considering ways to open up our own repository for people to explore. Part of the technology partner role we play for publishers is ensuring that their content reaches the widest audience and is accessible as possible. With a growing number of applications exposing Semantic Web data, access to RDF and SPARQL interfaces will be just as important as "traditional" and community specific interfaces such as OpenURL, and ZING. Historically there's been some disconnects between the library and publisher communities on how they store, manage and share metadata. As we're firmly in-between these communities I'm hoping we can start to help bridge these gaps, and I'm convinced that Semantic Web technologies can help.
posted by Leigh Dodds at 11:16 am
FOAF for the lazy, with embedded RDF
Thursday, May 25, 2006
So, in the New World Order... all documents and things on the web must have machine readable RDF Metadata. But, I'd never until this week actually bothered to produce some DC+ metadata for our project pages, or a FOAF file for the team, or myself.
Why? Well:
1. Who the heck wants to write machine readable metadata?
- I mean, seriously, tapping out angle brackets in notepad is grim enough if you have to write a bit of HTML, at least then you get to hit refresh and see something shiny. But writing RDF/XML ? Come on.
2. I'm too lazy to keep it up to date.
- It will become yet another piece of internet junk, caused by me.
3. There's something faintly embarassing about it.
- The prospect of writing a FOAF file gives me the shivers - like writing a CV. A 'homepage' with a tiled background and a picture of my hamster/holiday - now that would be fun. But a FOAF file? I suspect it's supposed to be rather more swish and work-ish with an impressive list of things I've done and VIPs I've met. Eww.
So I was quite keen on Ian from Talis's talk about embedded RDF at XTech . Basically the idea is to write an HTML page first - so the prose is aimed at humans - and then add a few extra tags and attrs to the HTML to define the things that would be needed to make some RDF metadata. Then you use some XSLT to scrape it out on the fly - there's even a webservice to do the transform for you.
My efforts mainly consisted of putting 'rel' attributes in my existing
Eg:
becomes
-> creates the triple:
And:
becomes
-> creates the triple:
Obviously there's much more to it than this. Eg, how to get
But I just got to grips with the basics. I was able to make this foaf metadata from my new homepage.
The con is:
The *extra stuff* in your HTML does make it a bit crowded.
The pros are:
1. It's lo-fi. Just a text editor and basic HTML skills required to get going - people may even do it.
2. It gets kept up to date automatically, as you update the human-readable page - which you *might* do.
3. You don't have type out RDF/XML.
4. Less ewww.
Now I've been inspired to produce this nice machine readable page, I can query it with Leigh's newly extended SPARQL service, to get results like this.
Why? Well:
1. Who the heck wants to write machine readable metadata?
- I mean, seriously, tapping out angle brackets in notepad is grim enough if you have to write a bit of HTML, at least then you get to hit refresh and see something shiny. But writing RDF/XML ? Come on.
2. I'm too lazy to keep it up to date.
- It will become yet another piece of internet junk, caused by me.
3. There's something faintly embarassing about it.
- The prospect of writing a FOAF file gives me the shivers - like writing a CV. A 'homepage' with a tiled background and a picture of my hamster/holiday - now that would be fun. But a FOAF file? I suspect it's supposed to be rather more swish and work-ish with an impressive list of things I've done and VIPs I've met. Eww.
So I was quite keen on Ian from Talis's talk about embedded RDF at XTech . Basically the idea is to write an HTML page first - so the prose is aimed at humans - and then add a few extra tags and attrs to the HTML to define the things that would be needed to make some RDF metadata. Then you use some XSLT to scrape it out on the fly - there's even a webservice to do the transform for you.
My efforts mainly consisted of putting 'rel' attributes in my existing
<a href>nchors, and adding a few extra <span>s. Eg:
<p>My name is Katie.</p>becomes
<p>My name is <span class="foaf-firstName">Katie</span></p>-> creates the triple:
_x foaf:firstName Katie And:
<p>I work at <a href="http://www.ingenta.com/corporate/">Ingenta</a></p>becomes
<p>I work at <a href="http://www.ingenta.com" rel="foaf-workplaceHomepage">Ingenta</a></p>-> creates the triple:
_x foaf:workplaceHomepage http://www.ingenta.comObviously there's much more to it than this. Eg, how to get
rdf:types in there.But I just got to grips with the basics. I was able to make this foaf metadata from my new homepage.
The con is:
The *extra stuff* in your HTML does make it a bit crowded.
The pros are:
1. It's lo-fi. Just a text editor and basic HTML skills required to get going - people may even do it.
2. It gets kept up to date automatically, as you update the human-readable page - which you *might* do.
3. You don't have type out RDF/XML.
4. Less ewww.
Now I've been inspired to produce this nice machine readable page, I can query it with Leigh's newly extended SPARQL service, to get results like this.
posted by Katie Portwin at 5:02 pm
It's worth letting your techies out every now and then...
Saturday, May 20, 2006
Congrats to Katie and Priya, who presented a paper on the Metastore project at the Hewlett Packard Jena User Conference earlier this month, and came away with the prize for "Best Applications Paper".
They went on to talk further about the project at XTech in Amsterdam, where Leigh also presented a paper on SPARQL.
Watch this space for further updates on these and other projects...
They went on to talk further about the project at XTech in Amsterdam, where Leigh also presented a paper on SPARQL.
Watch this space for further updates on these and other projects...
posted by Kirsty Meddings at 1:49 pm
Subscribe to this site
