February 24, 2010
Riak has a notion of “links” as part of the metadata of its objects. We talk about traversing, or “walking”, links, but what do the queries for doing so actually look like?
Let's put four objects in riak:
|
|
$ curl -X PUT -H "content-type:text/plain" \ -H "Link: </riak/hb/second>; riaktag=\"foo\", </riak/hb/third>; riaktag=\"bar\"" \ http://localhost:8098/riak/hb/first --data "hello" $ curl -X PUT -H "content-type: text/plain" \ -H "Link:</riak/hb/fourth>; riaktag=\"foo\"" \ http://localhost:8098/riak/hb/second --data "the second" $ curl -X PUT -H "content-type: text/plain" \ -H "Link:</riak/hb/fourth>; riaktag=\"foo\"" \ http://localhost:8098/riak/hb/third --data "the third" $ curl -X PUT -H "content-type: text/plain" \ http://localhost:8098/riak/hb/fourth --data "the fourth"
Now, say we wanted to start at hb/first, and follow
all of its outbound links. The easiest way to do this is with the
link-walker URL syntax:
$ curl http://localhost:8098/riak/hb/first/_,_,_
The response will be a multipart/mixed body with two parts: the
hb/second object in one, and the hb/third object in the
other:
--N2gzGP3AY8wpwdQY0jio62L9nJm Content-Type: multipart/mixed; boundary=3ai6VRl4aLli3dKw8tG9unUeznT --3ai6VRl4aLli3dKw8tG9unUeznT X-Riak-Vclock: a85hYGBgzGDKBVIsTKLLozOYEhnzWBn+H/h5hC8LAA== Location: /riak/hb/third Content-Type: text/plain Link: </riak/hb>; rel="up", </riak/hb/fourth>; riaktag="foo" Etag: 5Fs0VskZWx7Y25tf1oQsvS Last-Modified: Wed, 24 Feb 2010 15:25:51 GMT the third --3ai6VRl4aLli3dKw8tG9unUeznT X-Riak-Vclock: a85hYGBgzGDKBVIsLEHbN2YwJTLmsTLMPvDzCF8WAA== Location: /riak/hb/second Content-Type: text/plain Link: </riak/hb>; rel="up", </riak/hb/fourth>; riaktag="foo" Etag: 2ZKEJ2gaT57NT7xhLDPCQz Last-Modified: Wed, 24 Feb 2010 15:24:11 GMT the second --3ai6VRl4aLli3dKw8tG9unUeznT-- --N2gzGP3AY8wpwdQY0jio62L9nJm--
It's also possible to express the same query in map-reduce, directly:
$ curl -X POST -H "content-type:application/json" \
http://localhost:8098/mapred --data @-
{"inputs":[["hb","first"]],"query":[{"link":{}},{"map":{"language":"javascript","source":"function(v)
{ return [v]; }"}}]}
^D
That's the exact same query. The content type of the response is
different. It's now a JSON array with two elements: a JSON encoding
of the hb/second object, and a JSON encoding of
the hb/third object. (Pretty-printed here, for
clarity.)
[
{
"bucket": "hb",
"key": "second",
"vclock": "a85hYGBgzGDKBVIsLEHbN2YwJTLmsTLMPvDzCF8WAA==",
"values": [
{
"metadata": {
"Links": [
["hb","fourth","foo"]
],
"X-Riak-VTag": "2ZKEJ2gaT57NT7xhLDPCQz",
"content-type": "text/plain",
"X-Riak-Last-Modified": "Wed, 24 Feb 2010 15:24:11 GMT",
"X-Riak-Meta": []
},
"data": "the second"
}
]
},
{
"bucket": "hb",
"key": "third",
"vclock": "a85hYGBgzGDKBVIsTKLLozOYEhnzWBn+H/h5hC8LAA==",
"values": [
{
"metadata": {
"Links": [
["hb","fourth","foo"]
],
"X-Riak-VTag": "5Fs0VskZWx7Y25tf1oQsvS",
"content-type": "text/plain",
"X-Riak-Last-Modified": "Wed, 24 Feb 2010 15:25:51 GMT",
"X-Riak-Meta": []
},
"data": "the third"
}
]
}
]
Another interesting query is “follow only links that are tagged
foo.” For that, just add a tag field
to the link phase spec:
$ curl -X POST -H "content-type:application/json" \
http://localhost:8098/mapred --data @-
{"inputs":[["hb","first"]],"query":[{"link":{"tag":"foo"}},{"map":{"language":"javascript","source":"function(v)
{ return [v]; }"}}]}
^D
Here you should get a JSON array with one element: a JSON encoding
of the hb/second object. The link to
the hb/third object was tagged bar, so that
link was not followed. The equivalent URL syntax is:
$ curl http://localhost:8098/riak/hb/first/_,foo,_
It's also possible to filter links by bucket by adding a
bucket field to the link phase spec, or by replacing the
first underscore with a bucket name in the URL format. But, all of
our example links point to the same bucket, so
hb is the only interesting setting here.
Link phases may also be chained together (or put after other phases
if those phases produce bucket/key lists). For example, we could
follow the links all the way from hb/first
to hb/fourth with:
$ curl -X POST -H "content-type:application/json" \
http://localhost:8098/mapred --data @-
{"inputs":[["hb","first"]],"query":[{"link":{}},{"link":{}},{"map":{"language":"javascript","source":"function(v)
{ return [v]; }"}}]}
^D
(Notice the added link phase.) If you run that, you'll find that
you get two copies of the hb/fourth object in the
response. This is because we didn't bother uniquifying the results of
the link extraction, and both hb/second
and hb/third link to hb/fourth. A reduce
phase is fairly easy to add:
$ curl -X POST -H "content-type:application/json" \
http://localhost:8098/mapred --data @-
{"inputs":[["hb","first"]],"query":[{"link":{}},{"link":{}},{"reduce":{"language":"erlang","module":"riak_mapreduce","function":"reduce_set_union"}},{"map":{"language":"javascript","source":"function(v)
{ return [v]; }"}}]}
^D
The resource handling the URL link-walking format does just this:
$ curl http://localhost:8098/riak/hb/first/_,_,_/_,_,_
That should get you just one copy of the hb/fourth
object.
So why choose either map/reduce or URL-syntax? The advantage of URL syntax is that if you're starting from just one object, and just want to get the objects at the ends of the links, and you can handle multipart/mixed encoding, then URL syntax is much simpler and more compact. Map/reduce with link phases should be your choice if you want to start from multiple objects at once, or you want to get some processed or aggregated form of the objects, or you want the result to be JSON-encoded.
Riak version 0.8 note: In Riak 0.8, the format of the result of 'link' map/reduce phases was not able to be transformed into JSON. This meant both that it was not possible to put a Javascript reduce phase right after a link phase, and also that it was not possible to end an HTTP map/reduce query with a link phase. Those issues have been resolved in the tip of the source repository, and will be part of the 0.9 release.