clj-bookmarks

0.1.0


A client library for for bookmarking services such as del.icio.us or Pinboard. It provides both annonymous and named access. In the former case the APIs do not allow as many features as in the latter one.

dependencies

org.clojure/clojure
1.2.0
org.clojure/clojure-contrib
1.2.0
clj-http
0.1.3
midje
0.9.0

dev dependencies

marginalia
0.3.0



(this space intentionally left blank)
 

The core namespace provides the common API for the different bookmarking services.

(ns clj-bookmarks.core
  )

The AnonymousBookmarkService protocol

The common functions for accessing a bookmark service anonymously provided through the AnonymousBookmarkService protocol.

(defprotocol AnonymousBookmarkService
  (bookmarks [srv opts]
	     "The `bookmarks` function retrieves bookmarks from the
	     service according to criteria specified in `opts`:

             * `user`: only return bookmarks saved by this user
             * `tags`: only return bookmarks tagged with an element of this
                       seq

             At least one option must be specified.")
  (popular [srv]
	   "To get the most popular bookmarks from the service, call
           `popular`.")
  (recent [srv]
	  "To get the most recent bookmarks saved to the service by
          all users, call `recent`."))

The AuthenticatedBookmarkService protocol

The common functions that implement calls to the different bookmarking APIs are bundled in the BookmarkService protocol.

(defprotocol AuthenticatedBookmarkService
  (query-bookmarks [srv opts]
	     "The `query-bookmarks` function retrieves bookmarks from the
	     service according to criteria specified in `opts`:

             * `user`: only return bookmarks saved by this user
             * `tags`: only return bookmarks tagged with an element of this
                       seq
             * `fromdt`: only return bookmarks saved on this date or later
             * `todt`: only return bookmarks saved on this date or earlier
             * `offset`: start returning bookmarks this many results into
                         the set
             * `limit`: return this many results

             When called with an empty `opts` map, `bookmarks` returns
             the 15 most recent bookmarks saved by the user.")
  (add-bookmark [srv url desc opts]
		"Bookmarks can be added by calling `add-bookmark` for
                the service, passing a URL, a description and a map of
                options that can have the following keys (none of them
                required):

                * `tags`: a seq of tags
                * `date`: the datestamp assigned to the bookmark
                          (default: now)
                * `shared`: make the item public (default: true)
                * `replace`: replace a bookmark if the given URL has
                             already been saved (default: true)")
  (delete-bookmark [srv url]
		   "The `delete-bookmark` function deletes the bookmark
                   for the given URL.")
  (suggested-tags [srv url]
		  "You can get a vector of suggestions for tags for a
                  given URL from the service using the
                  `suggested-tags` function.")
  (bookmark-info [srv url]
		 "Call `bookmark-info` to retrieve the bookmark
                 structure for `url`.")
  (last-update [srv]
		"Use `last-update` to find the timestamp when the user
		last updated his bookmarks."))

The Bookmark Structure

The functions that return bookmarks always give you a map with the following structure:

  • url: the bookmarked URL
  • tags: the tags assigned to this bookmark
  • desc: the description
  • extended: Notes about the bookmark
  • date: the time when the bookmark was saved
  • others: the number of users who bookmarked the URL
  • hash: the hash of the URL
  • meta: a signature that changes when the bookmark is updated

Only url is always set. Functions from the anonymous APIs never set extended, others, hash, and meta.

 

The delicious namespace provides the implementation of the Delicious API.

(ns clj-bookmarks.delicious
  (:use [clj-bookmarks core util])
  (:require [clj-http.client :as http]
	    [clojure.contrib.zip-filter.xml :as zfx]
	    [clojure.contrib.zip-filter :as zf]
	    [clojure.string :as string])
  (:import [java.util Date Locale]
	   [java.text SimpleDateFormat]))

The Delicious v1 API

The following functions implement version 1 of the Delicious API (which is also used by Pinboard).

Parser Functions

The following functions parse the data returned from the service.

Parse a string of XML data containg a list of posts into the bookmark structure.

We first parse the data into a zipper, and extract the attributes of the post elements.

(defn parse-posts
  [input]
  (zfx/xml-> (str->xmlzip input) :post
	    (fn [loc] {:url (zfx/attr loc :href)
		       :tags (parse-tags (zfx/attr loc :tag))
		       :hash (zfx/attr loc :hash)
		       :meta (zfx/attr loc :meta)
		       :desc (zfx/attr loc :description)
		       :extended (zfx/attr loc :extended)
		       :date (parse-date (zfx/attr loc :time))})))

Parse a string of XML data with a response code from the Delicious v1 API and either return true or throw an exception.

The function returns true, when the code attribute equals done. Otherwise a exception is thrown with the code as message.

(defn parse-result
  [input]
  (let [code (zfx/xml1-> (str->xmlzip input) (zfx/attr :code))]
    (if-not (= code "done")
      ; FIXME a better error concept, maybe?
      (throw (Exception. code))
      true)))

Parse a string of XML data into a seq of tags.

We turn the data into a zipper and get the text of all leaf nodes.

(defn parse-suggestions
  [input]
  (zfx/xml-> (str->xmlzip input) zf/children zfx/text))

Parse a string of XML data into the date of the last modification.

We turn the input into a zipper and parse the date from the time attribute.

(defn parse-update
  [input]
  (parse-date (zfx/xml1-> (str->xmlzip input) (zfx/attr :time))))

Request Functions

The functions in the section send requests to the service with parameters formatted appropriately. The results are parsed using the functions from the previous section.

Query bookmarks from the Delicious v1 API.

This function sends a GET request to posts/all and appends some parameters as query-string. As we want to provide a convenient way to pass the parameters to this function, we need to convert the input opts before passing them on:

  • user: can be passed on as-is
  • tags: needs to be called tag for the request. Also a seq of tags must be converted to a space-delimited list.
  • fromdt: a Date object needs to be formatted
  • todt: same here
  • offset: called start in the API
  • limit: called results in the API

    The body of the response gets parsed into a seq of bookmark structures.

(defn posts-all
  [{:keys [endpoint] :as srv} opts]
  (letfn [(opt->param
	   [[k v]]
	   (cond
	    (= k :tags) [:tag (if (coll? v) (string/join " " v) v)]
	    (= k :limit)          [:results v]
	    (= k :offset)         [:start v]
	    (isa? (class v) Date) [k (format-date v)]
	    :else [k v]))]
    (let [params (into {} (map opt->param opts))]
      (-> (basic-auth-request srv (str endpoint "posts/all") params)
	  :body
	  (parse-posts)))))

Send a GET request to add a new bookmark with the Delicious v1 API.

First, we need to convert opts again; tags into a space-delimited string, date from a Date object to a string, and boolean values shared and replace to "yes" or "no". The options are all optional, but the bookmark must get a URL and a description.

Finally, we send the GET request, parse the result and raise an exception when something went wrong.

(defn posts-add
  [{:keys [endpoint] :as srv}
  url desc opts]
  (letfn [(opt->param
	   [[k v]]
	   (cond
	    (= k :tags) [:tags (if (coll? v) (string/join " " v) v)]
	    (isa? (class v) Date) [k (format-date v)]
	    (= k :shared)  [:shared  (if v "yes" "no")]
	    (= k :replace) [:replace (if v "yes" "no")]
	    :else [k v]))]
    (let [params
	  (into {:url url :description desc} (map opt->param opts))]
      (-> (basic-auth-request srv (str endpoint "posts/add") params)
	  :body
	  parse-result))))

Get the bookmark structure for url.

Send a GET request to posts/get with the URL as query parameter, and than parse the response body to yield a list with zero or one bookmarks. Finally, call first to return either nil or the bookmark.

(defn posts-get
  [{:keys [endpoint] :as srv} url]
  (-> (basic-auth-request srv (str endpoint "posts/get") {:url url})
      :body
      parse-posts
      first))

Delete a bookmark using the Delicious v1 API.

Send a GET request to posts/delete with the URL as query parameter. The response body is parsed and we return true when it worked and throw an exception otherwise.

(defn posts-delete
  [{:keys [endpoint] :as srv} url]
  (-> (basic-auth-request srv
			  (str endpoint "posts/delete") {:url url})
      :body
      parse-result))

Retrieve a list of suggested tags for a given URL using the Delicious v1 API.

We send a GET request with the URL as query parameter to posts/suggestand parse the result body to get the seq of tags.

(defn posts-suggest
  [{:keys [endpoint] :as srv} url]
  (-> (basic-auth-request srv
			  (str endpoint "posts/suggest") {:url url})
      :body
      parse-suggestions))

Get the time of the last modification of an account using the Delicious v1 API.

We send a GET request to posts/update and parse the result body into a Date object.

(defn posts-update
  [{:keys [endpoint] :as srv}]
  (-> (basic-auth-request srv (str endpoint "posts/update") {})
      :body
      parse-update))

The DeliciousV1Service Record

DeliciousV1Service implements the AuthenticatedBookmarkService protocol for the Delicious v1 API. Requires authentication (username, password).

Internally the implementation is delegated to the functions above.

(defrecord DeliciousV1Service [endpoint user passwd]
  AuthenticatedBookmarkService
  (query-bookmarks [srv opts] (posts-all srv opts))
  (add-bookmark [srv url desc opts] (posts-add srv url desc opts))
  (bookmark-info [srv url] (posts-get srv url))
  (delete-bookmark [srv url] (posts-delete srv url))
  (suggested-tags [srv url] (posts-suggest srv url))
  (last-update [srv] (posts-update srv)))

The Delicious RSS Feeds

The functions here are responsible for getting data out of the Delicious RSS feeds.

(def *del-base-rss-url* "http://feeds.delicious.com/v2/rss/")

Parser Functions

Create a SimpleDateFormat object for the format used by the Delicious RSS feeds.

As the format uses names for weekday and months, we need to set the locale to US.

(defn rss-date-format
  []
  (SimpleDateFormat. "EEE',' dd MMM yyyy HH:mm:ss Z" (Locale/US)))

Parse a date string in the format used by the Delicious RSS feeds into a Date object.

(defn parse-rss-date
  [input]
  (.parse (rss-date-format) input))

Parse a string of RSS data from Delicious into a list of posts.

The input is turned into a zipper which we use to extract the data from the item elements. The fields we need are in sub-elements:

  • link: put verbatimly into url in the result
  • category: these are the tags which we gather into a vector
  • title: we call this desc
  • pubDate: this is parsed into a Date object and called date.
(defn parse-rss-posts
  [input]
  (zfx/xml-> (str->xmlzip input) :channel :item
	     (fn [loc] {:url (zfx/xml1-> loc :link zfx/text)
			:tags (vec
			       (zfx/xml-> loc :category zfx/text))
			:desc (zfx/xml1-> loc :title zfx/text)
			:date (parse-rss-date
				  (zfx/xml1-> loc :pubDate zfx/text))})))

Request Functions

Get the currently popular bookmars using the Pinboard RSS feeds.

We send a GET request to popular and parse the response body into a seq of bookmarks.

(defn rss-popular
  []
  (-> (http/get (str *del-base-rss-url* "popular/"))
      :body
      parse-rss-posts))

Get the recent bookmars using the Delicious RSS feeds.

We send a GET request to recent and parse the response body into a seq of bookmarks.

(defn rss-recent
  []
  (-> (http/get (str *del-base-rss-url* "recent/"))
      :body
      parse-rss-posts))

The rss-bookmarks function uses the RSS feeds to perform a query for shared bookmarks.

The parameter map can include tags and user. tags can be either a string or a vector of string. The map must not be empty.

This function sends a GET request with the path /USER/TAG+TAG or /tag/TAG+TAG when no user is specified.

(defn rss-bookmarks
  [{:keys [tags user]}]
  (let [tags (if (string? tags) tags (string/join "+" tags))
	path (string/join "/" (filter (comp not string/blank?)
				      [(or user "tag") tags]))]
    (-> (http/get (str *del-base-rss-url* path))
	:body
	parse-rss-posts)))

The DeliciousRSSService Record

DeliciousRSSService implements the BookmarkService protocol for the Delicious RSS feeds. No authentication is required.

(defrecord DeliciousRSSService []
  AnonymousBookmarkService
  (bookmarks [srv opts] (rss-bookmarks opts))
  (popular [srv] (rss-popular))
  (recent [srv] (rss-recent)))
(def *del-base-api-url* "https://api.del.icio.us/v1/")

Create a service handle for Delicious.

When called without arguments, the RSS API is used.

When you pass a username and password, the full API is used.

(defn init-delicious
  ([] nil)
  ([user passwd] (init-delicious *del-base-api-url* user passwd))
  ([endpoint user passwd] (DeliciousV1Service. endpoint user passwd)))
 

The pinboard namespace provides the implementation of the Pinboard API.

(ns clj-bookmarks.pinboard
  (:use [clj-bookmarks core util])
  (:require [clj-http.client :as http]
	    [clojure.contrib.zip-filter.xml :as zfx]
	    [clj-bookmarks.delicious :as del]
	    [clojure.string :as string])
  (:import [java.util TimeZone Date]
	   [java.text SimpleDateFormat]))
(def *pb-base-api-url* "https://api.pinboard.in/v1/")

The Pinboard RSS Feeds

The functions here are responsible for getting data out of the Pinboard RSS feeds.

(def *pb-base-rss-url* "http://feeds.pinboard.in/rss/")

Parser Functions

Create a SimpleDateFormat object for the format used by the Pinboard RSS feeds.

The format object needs to be set to the UTC timezone, otherwise it would use the default timezone of the current machine. The dates in the RSS do provide a timezone, but it is always UTC and it is formatted in a way that SimpleDateFormat does not seem to support: In the feed there is an appended +00:00, but we could only parse either GMT+00:00 or +0000.

(defn rss-date-format
  []
  (doto (SimpleDateFormat. "yyyy-MM-dd'T'HH:mm:ss")
    (.setTimeZone (TimeZone/getTimeZone "UTC"))))

Parse a date string in the format used by the Pinboard RSS feeds into a Date object.

(defn parse-rss-date
  [input]
  (.parse (rss-date-format) input))

Parse a string of RSS data from Pinboard into a list of posts.

The input is turned into a zipper which we use to extract the data from the item elements. The fields we need are in sub-elements:

  • link: put verbatimly into url in the result
  • dc:subject: these are the tags which we split into a vector
  • description: we call this desc
  • dc:date: this is parsed into a Date object and called date.
(defn parse-rss-posts
  [input]
  (zfx/xml-> (str->xmlzip input) :item
	    (fn [loc] {:url (zfx/xml1-> loc :link zfx/text)
		       :tags (parse-tags
			      (zfx/xml1-> loc :dc:subject zfx/text))
		       :desc (zfx/xml1-> loc :description zfx/text)
		       :date (parse-rss-date
			      (zfx/xml1-> loc :dc:date zfx/text))})))

Request Functions

Get the currently popular bookmars using the Pinboard RSS feeds.

We send a GET request to popular and parse the response body into a seq of bookmarks.

(defn rss-popular
  []
  (-> (http/get (str *pb-base-rss-url* "popular/"))
      :body
      parse-rss-posts))

Get the recent bookmars using the Pinboard RSS feeds.

We send a GET request to recent and parse the response body into a seq of bookmarks.

(defn rss-recent
  []
  (-> (http/get (str *pb-base-rss-url* "recent/"))
      :body
      parse-rss-posts))

The rss-bookmarks function uses the RSS feeds to perform a query for shared bookmarks.

The parameter map can include tags and user. tags can be either a string or a vector of string. The map must not be empty.

This function sends a GET request with the path /u:USER/t:TAG/t:TAG.

(defn rss-bookmarks
  [{:keys [tags user]}]
  (let [tags (if (string? tags) [tags] tags)
	path (string/join "/" (filter (comp not nil?)
				      (cons  (if user (str "u:" user))
					     (map #(str "t:" %) tags))))]
    (-> (http/get (str *pb-base-rss-url* path))
	:body
	parse-rss-posts)))

The PinboardRSSService Record

PinboardRSSService implements the BookmarkService protocol for the Pinboard RSS feeds. No authentication is required.

(defrecord PinboardRSSService []
  AnonymousBookmarkService
  (bookmarks [srv opts] (rss-bookmarks opts))
  (popular [srv] (rss-popular))
  (recent [srv] (rss-recent)))

Create a service handle for Pinboard.

When called without arguments, the RSS feeds are used.

When you pass a username and password, the API (which is modeled on the Delicious API) is used.

(defn init-pinboard
  ([] (PinboardRSSService.))
  ([user passwd] (del/init-delicious *pb-base-api-url* user passwd)))
 

Common utility functions live in the 'util' namespace.

(ns clj-bookmarks.util
  (:require [clj-http.client :as http]
	    [clojure.xml :as xml]
	    [clojure.zip :as zip]
	    [clojure.contrib.zip-filter.xml :as zf]
	    [clojure.contrib.zip-filter :as zfilter]
	    [clojure.string :as string])
  (:import [java.util TimeZone Date]
	   [java.text SimpleDateFormat]
	   [java.io StringReader]
	   [org.xml.sax InputSource]))

Covert a string into a SAX InputSource.

In order to parse XML data from a string, you need to put it into an InputSource object. When you pass a string to clojure.xml/parse it interprets it as a filename.

(defn string->input-source
  [s]
  (InputSource. (StringReader. (.trim s))))

Turn a string of XML data into a zipper structure.

(defn str->xmlzip
  [input]
  (-> input
      string->input-source
      xml/parse
      zip/xml-zip))

Create a SimpleDateFormat object for the format used by the Delicious v1 API.

The format object needs to be set to the UTC timezone, otherwise it would use the default timezone of the current machine.

(defn date-format
  []
  (doto (SimpleDateFormat. "yyyy-MM-dd'T'HH:mm:ss'Z'")
    (.setTimeZone (TimeZone/getTimeZone "UTC"))))

Convert a Date object into a string with the format expected by the Delicious v1 API.

(defn format-date
  [d]
  (.format (date-format) d))

Parse a date string in the format used by the Delicious v1 API into a Date object.

(defn parse-date
  [input]
  (.parse (date-format) input))

Send an HTTP GET request using basic authentication to the given URL to which params get attached.

The first argument is a map with the keys user and passwd used for the authentication (usually the service handle records are used here).

(defn basic-auth-request
  [{:keys [user passwd]} url params]
  (http/get url {:query-params params
		 :basic-auth [user passwd]}))

Parse a space delimited string of tags into a vector.

(defn parse-tags
  [input]
  (vec (string/split input #"\s")))