In this post I will develop a small demo, showing how to write digital ink/pen software using Clojure, OpenGL (in the form of JOGL) and JPen. The end result will look simply like this:
Preliminary rambling
I love writing with a pen and I have long been enthusiastic about pen-based computing. In fact, I do most of my writing with Xournal. And with tablet computers becoming mainstream one might think that I am now getting all the innovative pen-centric applications I have always hoped for, right? Not so.
While multitouch tablets like the iPad are all the rage right now, pen-based tablet are woefully neglected, even though they have been around much longer. When I say pen-based, I mean tablets that have an active, pressure-sensitive digitizer as produced, for example, by WACOM. Tablet PCs with an active digitizer have been around long before multitouch technology was popularized by the iPhone. Yet, tablet PCs have never been able to catch on and have never received as much attention. I can only speculate what the reasons for this are. They may include the following.
First, Microsoft, the main proponent of tablet PCs, failed utterly to produce a pen-centric user interface. Then “pen” or “ink” features of Windows always felt like an add-on. Users always remained woefully aware, that for most purposes their pen functioned just like a mouse. The issues that arise range from such fundamentals as the fact that pens generate strokes not clicks to such details as the fact that menus expand down and to the right, so the hand of a right-handed user partially obscures the menu.
Second, Microsoft targeted tablet PCs at the wrong market. They tried to sell tablet PCs to managers and business people. But to this audience pens are of limited use – they are completely happy with typing meeting notes into a word processor. Creative people and scientists, on the other hand, find the ability to make quick hand-drawn sketches invaluable. But these audiences would have required an entirely different marketing campaign (as well as a more attractive user interface). It is a pity that Microsoft cancelled the production of its innovative Courier device.
Third, the price of tablet PCs was (and is) way too high. One reason for this was Microsoft’s targeting of business customers. Another may have been WACOMs long-time patent-enforced monopoly on the active digitizer market. This high-price policy made tablet PCs unattainable for creatives, students and scientists who are often short on funds.
Currently, it does not look like the situation is going to change. While there are a number of competitors to the iPad, I do not know of any that features an active pressure-sensitive pen. If you do, please drop me a line.
However, there are tablet PCs with active digitizers out there. And while current pen-centric software and user interfaces leave something to be desired, the technology is there to create new digital ink applications. So, let’s get to it!
Overview
In this blog post, I will develop an elementary digital ink application while exploring some technologies I am curious about. The goal is to create a simple drawing surface on which the user can write with a pen. The point is use the pressure sensitivity and subpixel precision of the pen to render high-quality pen strokes in real time. To this end we will use JPen to gather the pen data and OpenGL, as provided by JOGL, to render the strokes. As programming language we will use Clojure. Along the way we are going to explore a way to separate the application logic from all the library-wrangling we will have to do.
About the choice of tools: I am using Clojure simply because I always wanted to write software using a Lisp dialect. Its parallelism features will come in handy when exchanging data between the JPen thread and the OpenGL thread and its macro facilities will be helpful when separating logic from plumbing. JPen is hands-down the best pen library for Java. It binds to the platform specific pen APIs. Using OpenGL is probably a bit too much for such a simple application. However, its performance will useful and we will have the opportunity to learn how some of the work of drawing pen strokes can be shifted to the GPU. JOGL is one of several OpenGL bindings available, which I picked simply because I found it the most appealing. Finally, I chose the JVM as platform because I wanted to make use of pen and OpenGL libraries across several platforms without having to worry about calling native methods and recompiling.
What is the application supposed to do?
Before we dive in and try to get all the libraries to do what we want, let us take a step back and figure out what we want to do, exactly. Speaking with Fred Brooks, we want to determine the essential complexity of the program we are trying to write and worry about the accidental complexity later. To quote Moseley and Marks: “Essential Complexity is inherent in, and the essence of, the problem as seen by the users.”
In our case this essential complexity is how user input changes the state of the user interface. Nothing else. The reason for this is simply that, from the point of view of the user, the application does nothing else. It does not compute anything and its use does not have side-effects on the rest of the system.
The user interacts with the application solely through the pen. User input, therefore, is a sequence of tuples $(x,y,p)$ where $(x,y)$ is the position of the pen in window coordinates and $p\in[0,1]$ is the pressure of the pen on the tablet surface. (In fact, one could argue that this is not the input as the user sees it, as the user perceives pen motions as continuous. So by virtue of the fact that the tablet is making discrete measurements we have already introduced accidental complexity. We will disregard this complication, however.)
The purpose of the app is to present a writing surface to the user. At any point in time the surface contains a set of strokes. Each stroke is sequence of triples $(x,y,p)$ with $p>0$. However, if the pen currently rests on the surface, one stroke has a special role: subsequent motions of the pen extend this current stroke by adding triples, until the pen is lifted off the surface. We can thus represent the state of the application by a pair $(S,c)$ where $S$ is a set of strokes and $c$ is the current stroke or $\text{nil}$, indicating that there is no current stroke. Also, we may want to present a cursor to the user. This makes the current pen data $(x,y,p)$ also part of the state. This leads us to the application state $(x,y,p,S,c)$. Initially, this state is set to $(0,0,0,[],\text{nil})$.
How does the application state change upon the input of a new triple $(x^i,y^i,p^i)$? We are going to describe the new state $(x^n,y^n,p^n,S^n,c^n)$ in terms of the input and the old state $(x^o,y^o,p^o,S^o,c^o)$.
First of all we update the cursor. Next, there are four cases to consider.
- If the pen is on the tablet and the user was drawing a stroke, then we append the new data to the current stroke.
- If the pen is on the tablet but the user was not drawing a stroke, we start a new stroke.
- If the pen is off the tablet and the user was drawing a stroke, then we move the current stroke to our set of strokes.
- If the pen is off the tablet and the user was not drawing a stroke, we need not do anything beyond updating the cursor.
Formally, using $[]$ to denote lists and $\circ$ to denote concatenation, this can be described thus:
$$
\begin{array}{lclrcll}
&&& x^n &:=& x^i \cr
&&& y^n &:=& y^i \cr
&&& p^n &:=& p^i \cr
\text{ if } & p^i > 0 \wedge c^o \not= \text{nil} & \text{ then } & c^n & := & c^o \circ [(x^i,y^i,p^i)] \cr
\text{ if } & p^i > 0 \wedge c^o = \text{nil} & \text{ then } & c^n & := & [(x^i,y^i,p^i)] \cr
\text{ if } & p^i = 0 \wedge c^o \not= \text{nil} & \text{ then } & c^n &:=& \text{nil} \cr
&&& S^n &:=& S^o \cup \{c^o\}
\end{array}
$$
This determines completely what the application is supposed to do. In an ideal world where accidental complexity does not appear we would be done by now. In the real world, things are more complicated. Nonetheless, it is desirable to express the application logic of the program as simply as possible. This is where macros come in.
Transforming the UI state
The triples $(x,y,p)$ a stroke consists of will be represented in Clojure by records:
(defrecord StrokePoint [x y p])
Similarly, the state $(x,y,p,S,c)$ becomes
(defrecord AppState [pen-x pen-y pen-pres strokes current-stroke])
and is initialized by
(def the-state (ref (AppState. 0.0 0.0 0.0 (vector) (vector) )))
The state transformation we define thus:
(defstatetrafo process-input the-state [x y p]
(let [in-stroke (not (nil? (§ :current-stroke)))
number-of-strokes (count (§ :strokes))]
(:= :pen-x x :pen-y y :pen-pres p)
(cond
(and (> p 0.0) in-stroke)
(:=
:current-stroke (conj (§ :current-stroke) (StrokePoint. x y p)))
(and (> p 0.0) (not in-stroke))
(:=
:current-stroke (vector (StrokePoint. x y p)))
(and (= p 0.0) in-stroke)
(:=
:current-stroke nil
[:strokes number-of-strokes] (§ :current-stroke)))))
This is pleasantly close to the mathematical definition while still looking like Lisp code. Its semantics are as follows. Variables are represented by keywords :pen-x, :current-stroke and so on. Instead of using subscripts to distinguish the values of the variables in the old state from the values of the variables in the new state, we use, e.g., (§ :current-stroke) to refer to the variable :current-stroke in the old state.
The values of the variables in the new state cannot be read, they can only be written. To this end we use the syntax :=, which acts like a function that takes parameters $l_1,r_1,\ldots,l_k,r_k$ and assigns $l_i:=r_i$ for $i=1,\ldots,k$. Simply put, := acts like assoc, with the difference that we do not need to pass the data structure we are modifying around. Note that variables that are not given explict assignments take the same value in the new state as in the old state.
What still requires explanation is the appearance of [:strokes number-of-strokes] on the left-hand side of an assignment. If a vector appears on the left-hand side of an assignment it is interpreted as a path into a nested data structure, and the appropriate location is changed. (Think of assoc-in.) In this example, the assignment appends the current stroke to the end of the vector :strokes.
defstatetrafo is not standard Clojure, but a custom macro I have written. It takes a name, a reference to a state, a parameter vector and a body of expressions and is defined thus:
(defmacro defstatetrafo [name state params & body]
(let [old-state (gensym)
tmp-state (gensym)
state-updater (gensym)
body-rewriter (fn [tree]
(clojure.walk/postwalk
(fn [a]
(cond
(= a :=) state-updater
(and (list? a) (= (first a) (symbol "§"))) (concat `(-> ~old-state) (rest a))
:else a))
tree))]
`(defn ~name ~params
(let [~old-state (deref ~state)
~tmp-state (ref ~old-state)
~state-updater (fn [& args#]
(ref-set ~tmp-state (apply assoc-in-mult (deref ~tmp-state) args#)))]
(dosync ~@(map body-rewriter body))
(dosync (ref-set ~state (deref ~tmp-state)))))))
Instead of going through this line by line, let me just show you what the state transformer given above is rewritten into. (I have removed all namespace qualifications of the output to make it more readable.)
(defn process-input [x y p]
(let [G__3254 @the-state
G__3255 (ref G__3254)
G__3256 (fn [& args__3211__auto__]
(ref-set G__3255 (apply assoc-in-mult @G__3255 args__3211__auto__)))]
(dosync
(let [in-stroke (not (nil? (-> G__3254 :current-stroke)))
number-of-strokes (count (-> G__3254 :strokes))]
(G__3256 :pen-x x :pen-y y :pen-pres p)
(cond
(and (> p 0.0) in-stroke)
(G__3256 :current-stroke (conj (-> G__3254 :current-stroke) (StrokePoint. x y p)))
(and (> p 0.0) (not in-stroke))
(G__3256 :current-stroke (vector (StrokePoint. x y p)))
(and (= p 0.0) in-stroke)
(G__3256 :current-stroke nil [:strokes number-of-strokes] (-> G__3254 :current-stroke)))))
(dosync (ref-set the-state @G__3255))))
As you can see, this creates a reference to hold a temporary state. This temporary state is initialized to the current value of the-state and then gets updated incrementally using the function G__3256 while the body of the state transformer is run. The final value is then written to the reference holding the-state inside a transaction.
The function assoc-in-mult, used above, works like assoc-in but takes an arbitrary number of key value pairs.
(defn assoc-in-mult
([s] s)
([s k v]
(if (vector? k)
(assoc-in s k v)
(assoc s k v)))
([s k v & rest]
(apply assoc-in-mult (cons (assoc-in-mult s k v) rest))))
I am using defstatetrafo in one other project, and there this way of defining the application logic concisely and separating it from all the rest has been very helpful. It makes experimenting with UI behavior particularly easy. How scalable this approach is, however, time will have to tell. Already, rendering this kind of state with OpenGL efficiently presents some challenges, as we will see in the next section.
Drawing the state with OpenGL
In this section, I will explain how drawing strokes with OpenGL works. This is not an OpenGL tutorial. The purpose of this section is to give a reader who has seen OpenGL before an idea how to work with vertex buffers, how to draw strokes in a digital ink application and how all of this goes together with the abstract application state management I mentioned above. It you are looking for a tutorial introduction to modern OpenGL, I recommend this one.
Naive approach
When I last looked at OpenGL many years ago, it was typically used in immediate mode. That is, the display callback contained code that drew all the interface one element at the time and it was run, say, 30 times per second. This approach is ideally suited for the kind of separation of logic from layout that I had in mind: You just define a function that draws the state on screen. In this spirit, one might write something like the following:
(doseq [stroke (:strokes @the-state)]
(doseq [point-pair (partition 2 stroke))]
(let [this-point (nth point-pair 0)
prev-point (nth point-pair 1)
x1 (:x prev-point)
y1 (:y prev-point)
x2 (:x this-point)
y2 (:y this-point)]
(doto gl
(.glLineWidth (:p this-point))
(.glBegin (. GL2 GL_LINES))
(.glVertex3f x1 y1 0.0)
(.glVertex3f x2 y2 0.0)
(.glEnd))))
In this way, each stroke would be represented by consecutive line segments where the width of each line segment depends on the pressure of the pen at one of its vertices.
However, it turns out that even on a fast computer, this trivial code leads to a noticeable drop in frame rate after a couple of minutes writing with the pen. One problem is the change of line width after each segment, as glBegin is expensive. But the most important factor is that immediate mode is notoriously slow: The CPU should not be concerned with passing vertex data to the GPU for every single frame! Thus, immediate mode has fallen out of fashion in OpenGL programming. So much so, in fact, that it has been deprecated in OpenGL 3.0.
In the following, we are going to explore a better way of drawing strokes: using triangle strips and vertex buffer objects.
Representing strokes with triangle strips
The idea of drawing a single stroke as a set of line segments of various widths is appealingly simple and used in many applications such as Xournal. However, when OpenGL is your graphics library, this is not a good idea, for several reasons.
- OpenGL does not support line caps and joins. This causes unwanted gaps between segments and makes it hard to make strokes appear smooth.
- Many vendors do not support the line width attribute and render all lines with 1 pixel width. Since OpenGL 3.0, line widths other than 1.0 are officially not part of the specification.
Instead we will represent strokes as triangle strips. Let $q_1,\ldots,q_N$ be the sequence of points $q_i=(x_i,y_i)$ representing our stroke. To each point $q_i$ we will associate a pair of vertices $v_{i,1},v_{i,2}$ and use the vertex sequence $v_{1,1},v_{1,2},v_{2,1},v_{2,2},\ldots,v_{N,1},v_{N,2}$ to construct our triangle strip. The idea is to choose the $v_{i,j}$ such that $v_{i,1}-v_{i,2}$ is “perpendicular” to the stroke we are trying to represent. The length $||v_{i,1}-v_{i,2}||$ will then be scaled to reflect the pressure $p_i$ at $q_i$.
To that end we are going to first construct a “tangent” $t_i$ to our stroke at point $q_i$ by \[ t_i=\frac{p_i-p_{i-1}}{||p_i-p_{i-1}||}, \] rotate $t_i$ by $90^\circ$ to obtain a perpendicular unit vector $t_i^\perp$ and then define
\[
\begin{eqnarray}
v_{i,1} & := & q_i + p_i \cdot t_i^\perp \
v_{i,1} & := & q_i – p_i \cdot t_i^\perp
\end{eqnarray}
\]
where $p_i$ is the pressure applied at $q_i$. This approach has the added advantage that the width of the stroke is interpolated between consecutive points of the stroke.
To realize this in code, we first define some elementary linear algebra functions.
(defn plus [p1 p2] (StrokePoint. (+ (:x p1) (:x p2)) (+ (:y p1) (:y p2)) (:p p1))) (defn scale [s p] (StrokePoint. (* s (:x p)) (* s (:y p)) (:p p))) (defn minus ([p] (StrokePoint. (- (:x p)) (- (:y p)) (:p p))) ([p q] (plus p (minus q)))) (defn minusp [pair] (minus (nth pair 0) (nth pair 1))) (defn norm [p] (clojure.contrib.math/sqrt (+ (* (:x p) (:x p)) (* (:y p) (:y p))))) (defn normalize [p] (let [n (norm p)] (if (= n 0.0) nil (StrokePoint. (/ (:x p) n) (/ (:y p) n) (:p p))))) (defn rot90 [p] (StrokePoint. (- (:y p)) (:x p) (:p p)))
Next we want to construct the triangle strip with a function stroke-to-coord-data. The input is a sequence of StrokePoints and the output is a sequence of floats, containing the coordinates of the vertices of the triangle strip in one long flat list – that is the format expected by OpenGL. We are going to make use of two helper functions, points-to-tangents and stroke-point-to-strip-vertices. The former takes a sequence of points and returns a sequence of tangents. The latter takes a single point $q_i$ and returns the pair of triangle strip vertices $v_{i,1},v_{i,2}$.
(defn stroke-point-to-strip-vertices [pair]
(let [point (nth pair 0) ntangent (nth pair 1)]
(if (nil? ntangent)
[]
(let [orth (rot90 ntangent)
sorth (scale (* 1.3 (:p point)) orth)
p1 (minus point sorth)
p2 (plus point sorth)]
[(:x p1) (:y p1)
(:x p2) (:y p2)]))))
(defn points-to-tangents [points]
(let [pairs (partition 2 1 points)
diffs (map minusp pairs)
tangents (map normalize diffs)]
(concat [(first tangents)] tangents)))
(defn stroke-to-coord-data [stroke]
(let [tangents (points-to-tangents stroke)
p-t-pairs (map list stroke tangents)
nested-floats (map stroke-point-to-strip-vertices p-t-pairs)]
(flatten nested-floats)))
Creating vertex buffer objects
Now that we have done all the theoretical geometry, how do we interact with OpenGL? We need to store the array of floats in GPU memory so that we can make use of them when drawing later on. This is achieved by make-float-buffer.
(defn make-float-buffer [^GL2 gl2 target data usage]
(let [buffers (int-array 1)
float-buffer (FloatBuffer/wrap (float-array data))]
(. float-buffer rewind)
(let [count (. float-buffer remaining)
size (* count (Buffers/SIZEOF_FLOAT))]
(doto gl2
(.glGenBuffers 1 buffers 0)
(.glBindBuffer target (aget buffers 0))
(.glBufferData target size float-buffer usage))
[(aget buffers 0) count])))
First, glGenBuffer creates a buffer in GPU memory and stores its id in buffers. Then, we make the buffer active by calling glBindBuffer. Finally we call glBufferData to actually store that data on the GPU. In the above code, target specifies what kind of buffer we are dealing with, while usage specifies how we intend to use it. In our case target will be GL_ARRAY_BUFFER, while usage will be GL_STATIC_DRAW.
Application state and OpenGL state
The challenge is now to propagate changes in the application state to the OpenGL state. Why are going to make our life easy: Whenever a new stroke is added to the set $S$, we will create and fill a new buffer in GPU memory. The current stroke $c$ we will always draw in immediate mode. To keep track of all the buffer ids returned by the OpenGL server, we will create a new state variable called opengl-state
(defrecord OpenGLState [stroke-buffer]) (def opengl-state (ref (OpenGLState. (vector))))
We will check if opengl-state needs updating and apply changes if necessary using the state transformer update-opengl-state. Its inputs are the OpenGL object and the vector $S$ of strokes.
(defstatetrafo update-opengl-state opengl-state [gl strokes]
(let [number-of-strokes (count strokes)
number-of-strokes-buffered (count (§ :stroke-buffer))]
(if (> number-of-strokes number-of-strokes-buffered)
(doseq [stroke-index (range number-of-strokes-buffered number-of-strokes)]
(let [stroke (nth strokes stroke-index)
data (stroke-to-coord-data stroke)
[buffer-id buffer-size] (make-float-buffer gl (. GL GL_ARRAY_BUFFER) data (. GL GL_STATIC_DRAW))]
(:= [:stroke-buffer stroke-index] [buffer-id (/ buffer-size 2)] ))))))
This ensures that stroke-buffer always contains a list of pairs [buffer-id number-of-vertices], one for each stroke in $S$, where buffer-id is id of the corresponding stroke and number-of-vertices is the number of vertices of the corresponding triangle strip.
The OpenGL event listener
Now we can finally get to the OpenGL event listener that does the actual drawing.
(def gl-event-listener
(proxy [GLEventListener] []
(display [^javax.media.opengl.GLAutoDrawable drawable]
(let [^GL2 gl (.. drawable getGL getGL2)
mystate (dosync @the-state)
x (-> mystate :pen-x)
y (-> mystate :pen-y)]
(update-opengl-state gl (:strokes mystate))
(doto gl
(.glClear (. GL GL_COLOR_BUFFER_BIT))
(.glColor3f 0 0 0)
(.glBegin (. GL2 GL_QUADS))
(.glVertex3f (- x 2) (- y 2) 0)
(.glVertex3f (+ x 2) (- y 2) 0)
(.glVertex3f (+ x 2) (+ y 2) 0)
(.glVertex3f (- x 2) (+ y 2) 0)
(.glEnd))
(doseq [[buffer-id vertex-number] (:stroke-buffer @opengl-state)]
(doto gl
(.glBindBuffer (. GL GL_ARRAY_BUFFER) buffer-id)
(.glEnableClientState (. GL2 GL_VERTEX_ARRAY))
(.glVertexPointer (int 2) (int (. GL GL_FLOAT)) (int (* 2 (. Buffers SIZEOF_FLOAT))) (long 0))
(.glDrawArrays (. GL2 GL_TRIANGLE_STRIP) 0 vertex-number)
(.glDisableClientState (. GL2 GL_VERTEX_ARRAY))
(.glBindBuffer (. GL GL_ARRAY_BUFFER) 0)
))
(. gl glBegin (. GL2 GL_TRIANGLE_STRIP))
(if (:current-stroke mystate)
(doseq [[x y] (partition 2 (stroke-to-coord-data (:current-stroke mystate)))]
(. gl glVertex3f x y 0.0)))
(. gl glEnd)))
(displayChanged [drawable mode-changed device-changed])
(init [^javax.media.opengl.GLAutoDrawable drawable]
(let [gl (.. drawable getGL getGL2)]
(prn (. drawable getChosenGLCapabilities))
(doto gl
(.glClearColor 1.0 1.0 1.0 0.0)
(.glShadeModel (. GL2 GL_FLAT))
(.glEnable (. GL2 GL_MULTISAMPLE)))))
(reshape [^javax.media.opengl.GLAutoDrawable drawable x y width height]
(let [gl (.. drawable getGL getGL2)]
(doto gl
(.glMatrixMode (. GL2 GL_PROJECTION))
(.glLoadIdentity)
(.glOrtho 0 width height 0 -1 1)
(.glMatrixMode (. GL2 GL_MODELVIEW)))))))
Let us consider the init callback first, which is used to set up the OpenGL state before any drawing happens. Here, there does not happen much. We print the capabilities of the OpenGL surface we are working with to standard output, for debugging purposes, we set the background color to white and enable multisampling (antialiasing).
The reshape callback is called whenever the size of the window changes. We use parallel projection and set up the coordinate system such that the top left corner of the window corresponds to coordinates $(0,0)$, the $x$-axis extends to the right and the $y$-axis extends downwards. Things are normalized so that one unit in view coordinates corresponds to one pixel on screen.
The display callback does the actual drawing. First we update the OpenGL state as described above. Then we clear the screen, set the foreground color to black, draw the cursor as a single quad and then draw the strokes. First, we draw the strokes in $S$ by making use of the vertex buffer objects we constructed. This proceeds as follows.
For each stroke:
- We activate the buffer containing the stroke data using
glBindBuffer - We use
glEnableClientStateto tell the OpenGL server that its going to take the vertex data for the subsequentglDrawArrayscommand from the vertex buffer. - We use
glVertexPointerto tell the OpenGL server how to interpret the data in the vertex buffer. In particular, every vertex is given by 2 entries of the buffer, every entry is a float, to get to the next vertex the pointer into the buffer needs to be advanced by2 * SIZEOF_FLOATbytes and the server is supposed to start at the beginning of the array. glDrawArraysdoes the actual drawing of the given data. Here we specify that we want to draw a triangle strip and how many vertices the OpenGL server is supposed to read from the array.- Finally, we clean up after ourselves by calling
glDisableClientStateandglBindBufferwith a zero argument.
In the end, as already mentioned, the current stroke is drawn in immediate mode.
One important point to stress are the explicit casts to primitive types in the call to glVertexPointer. Without these neither the Clojure compiler nor run-timer reflection are able to figure out the correct function to call and the program terminates with an exception. The type hints ^javax.media.opengl.GLAutoDrawable and ^GL2 are also vital.
What also happens frequently in this regard is that even though the Clojure compiler cannot resolve the correct method at compile-time, reflection at run-time succeeds. Especially in OpenGL code, reflection incurs a terrible speed penalty, however, and looking for the source of these performance problems can take the novice (me) a lot of time. Setting
(set! *warn-on-reflection* true)
is the key to finding these problems early on.
Pen input
We have said so much already and still, we did not get to handling pen data! Fortunately, that is much simpler than dealing with OpenGL – all thanks to the powers of JPen-2. Our pen event listener is simply this:
(def pen-event-listener
(proxy [PenListener] []
(penLevelEvent [^PLevelEvent ev]
(let [level-to-tuple
(fn [^PLevel level]
(condp = (. level getType)
PLevel$Type/X {:x (. level value)}
PLevel$Type/Y {:y (. level value)}
PLevel$Type/PRESSURE {:p (. level value)}
{}))
initial-tuple (dosync {:x (:pen-x @the-state) :y (:pen-y @the-state) :p (:pen-pres @the-state)})
values (reduce merge initial-tuple (map level-to-tuple (. ev levels)))]
(process-input (:x values) (:y values) (:p values))))
(penButtonEvent [ev])
(penKindEvent [ev])
(penScrollEvent [ev])
(penTock [millis])))
The axes of the pen we are interested in are the $x$- and $y$-coordinates as well as the pressure. The key feature to explain here is that not every pen level event ev comes with all three coordinates. Some events may contain only the $x$-coordinate, if the $x$-coordinate is all that changed. Each coordinate is contained in its own level object. So we need to iterate over all (. ev levels) and specify default values to make sure values contains values for all three coordinates. After that, all we do is process the input using our state transformer.
One fine point worth mentioning is the expression PLevel$Type/X. The $ is used to access a nested class while the / is used to access a static member. In Java the same reads simply PLevel.Type.X. It took me quite a while before I found out that $ and / are the proper characters to replace those two periods with.
Tying everything together
We use AWT’s Frame and JOGL’s GLCanvas to get an OpenGL window up and running. We also attach JPen’s PenManager to the canvas to receive pen events and, of course, we add the two listeners we defined.
(defn go []
(let [frame (new Frame)
caps (new GLCapabilities (GLProfile/get GLProfile/GL2))
chooser (new DefaultGLCapabilitiesChooser)]
(. caps setDoubleBuffered true)
(. caps setHardwareAccelerated true)
(. caps setSampleBuffers true)
(. caps setNumSamples 4)
(let [gl-canvas (new GLCanvas caps chooser nil nil)
animator (new FPSAnimator gl-canvas 60)
pen-manager (new PenManager gl-canvas)]
(. gl-canvas addGLEventListener gl-event-listener)
(.. pen-manager pen (addListener pen-event-listener))
(. frame add gl-canvas)
(let [pixels (int-array 256)
image (. (Toolkit/getDefaultToolkit) createImage (new MemoryImageSource 16 16 pixels 0 16))
cursor (. (Toolkit/getDefaultToolkit) createCustomCursor image (new Point 0 0) "invisibleCursor")]
(. frame setCursor cursor))
(. frame setSize 300 300)
(. frame
(addWindowListener
(proxy [WindowAdapter] []
(windowClosing [event]
(. System exit 0)))))
(. animator add gl-canvas)
(. animator start)
(. frame show))))
(go)
Noteworthy are two pieces of this code. At the beginning we use GLCapabilities to specify what properties we would like our OpenGL context to have. In particular
(. caps setSampleBuffers true) (. caps setNumSamples 4)
is used in conjunction with the (.glEnable (. GL2 GL_MULTISAMPLE)) command in the init callback to turn on multisampling (4x antialiasing). While this works wonderfully on my desktop, my tablet pc unfortunately does not support this feature. Fortunately, thanks to KluDX it is easy to find out what OpenGL capabilities a given chipset supports.
The other interesting part is
(let [pixels (int-array 256)
image (. (Toolkit/getDefaultToolkit) createImage (new MemoryImageSource 16 16 pixels 0 16))
cursor (. (Toolkit/getDefaultToolkit) createCustomCursor image (new Point 0 0) "invisibleCursor")]
(. frame setCursor cursor))
which is used to hide the mouse cursor, when the pen hovers over the application window.
Running the program
To run the program, you need copies of Clojure, JOGL and JPen appropriate for your system as well as a recent JDK. I am on Windows, put the source code in a folder called src, the libraries in a folder called libs and run the program using the following batch file:
:: Change the following to match your paths set CLOJURE_DIR=.\libs\clojure-1.2 set CLOJURE_JAR=%CLOJURE_DIR%\clojure.jar set CONTRIB_JAR=%CLOJURE_DIR%\clojure-contrib.jar set JPEN_DIR=.\libs\jpen-2 set JPEN_JAR=%JPEN_DIR%\jpen-2.jar set JPEN_DLL_DIR=%JPEN_DIR% set JOGL_DIR=.\libs\jogl-2.0-win set JOGL_JARS=%JOGL_DIR%\jar\gluegen-rt.jar;%JOGL_DIR%\jar\jogl.all.jar;%JOGL_DIR%\jar\nativewindow.all.jar;%JOGL_DIR%\jar\newt.jar set JOGL_DLL_DIR=%JOGL_DIR%\lib :: set DEBUG=-Djogl.debug=all java -cp .;%CLOJURE_JAR%;%CONTRIB_JAR%;%JPEN_JAR%;%JOGL_JARS% -Djava.library.path=%JPEN_DLL_DIR%;%JOGL_DLL_DIR% %DEBUG% clojure.main src\pengl.clj -- %* pause
Conclusion
Developing pen-centric applications with open source tools and a functional language is possible and, in fact, quite pleasant. (During similar experiments with VisualStudio, F# and WPF I have run into bigger trouble.) The above demo is not as short as it could be, but it also has a few features that a “minimal example” would lack: A strict separation of the application logic from the library dependent part of the program. The use of the GPU to draw strokes. Smooth rendering of strokes using triangle strips.
Regarding the individual components of this toolchain, I can say the following. Clojure has been a joy to use. Thanks to Clojure, the synchronization between the pen and OpenGL threads has been a non-issue and macros allowed me to experiment with custom state handling strategies. The only problem with Clojure is figuring out the mapping between Clojure and Java which is not always easy to decipher (e.g., the use of $ for referring to nested classes). JPen was perfect. It did what a library is supposed to do: it stayed out of the way. JOGL, as an OpenGL implementation, worked also very well, even though it requires some practice to translate the C++/OpenGL examples one finds on the net (e.g., the use of FloatBuffers; when are type casts necessary and when not).
Working with OpenGL in itself, however, was the most work. Taking the first steps was easy enough, but moving beyond immediate mode required the largest effort that went into writing this demo. I learned a lot in the process and using the GPU directly to draw strokes is kind of cool. However, it may well turn out that writing this kind of low level code is simply too much work for small projects, even if the subject is simple 2D drawing.
As already mentioned, I am rather pleased with my experiments with the state transformer. However, if it is feasible to maintain this separation remains to be seen, especially when the drawing code gets more complex and more state has to be kept in GPU memory. Also, this method of transforming state may be too slow in the long run.
Which brings me to the next topic: Performance. The performance of this demo program is brilliant on my desktop system and good on my tablet pc. However, I am under the impression, that a full fledged journal application such as Xournal would be too slow on my tablet when coded in this style. Whether my naive handling of state is the problem or rather the overhead Clojure imposes I cannot tell, yet. When improving the speed of the demo, both (set! *warn-on-reflection* true) as well as the profiler in clojure.contrib.profile were very helpful.
Finally, there are several directions in which it would be interesting to extend this demo (apart from building real applications of course).
- There are alternatives to multisampling to render antialiased strokes. Use these to get antialiasing on Intel graphics chips (which often power tablet PCs).
- Is it possible to make the program really fast? Is Clojure a bottleneck for performance here? Do these tips for improving performance of Clojure code help?
glVertexPointerand friends are not the very latest OpenGL either. (Deprecated in 3.0.) Migrate the code toglVertexAttribPointeretc. Use vertex array objects to shift even more drawing code onto the GPU.- Use geometry shaders and tessellation to actually compute the geometry of the triangle strip on the GPU. Use adaptive subdivision on the GPU to produce smoother strokes. (Of course this is pointless on tablet GPUs.)
- Implement pan and zoom. Render the strokes into textures to improve performance in large documents. Enable MIP mapping.
- Extend the demo to support deletion of strokes. Can the above method of updating opengl-state be extended to handle this case effectively?
Now, without further ado, here comes the complete code:
(import '(java.awt Frame Toolkit Point)
'(java.awt.image MemoryImageSource)
'(java.awt.event WindowListener WindowAdapter)
'(javax.media.opengl GLEventListener GL GL2 GL3 GLCapabilities GLProfile DefaultGLCapabilitiesChooser)
'(javax.media.opengl.awt GLCanvas)
'(com.jogamp.opengl.util FPSAnimator)
'(java.nio FloatBuffer)
'(com.jogamp.common.nio Buffers)
'(jpen.event PenListener)
'(jpen PButtonEvent PenManager PKindEvent PLevelEvent PScrollEvent PLevel PLevel$Type))
(require 'clojure.walk 'clojure.contrib.math)
(set! *warn-on-reflection* true)
;; State Transformer
(defn assoc-in-mult
([s] s)
([s k v]
(if (vector? k)
(assoc-in s k v)
(assoc s k v)))
([s k v & rest]
(apply assoc-in-mult (cons (assoc-in-mult s k v) rest))))
(defmacro defstatetrafo [name state params & body]
(let [old-state (gensym)
tmp-state (gensym)
state-updater (gensym)
body-rewriter (fn [tree]
(clojure.walk/postwalk
(fn [a]
(cond
(= a :=) state-updater
(and (list? a) (= (first a) (symbol "§"))) (concat `(-> ~old-state) (rest a))
:else a))
tree))]
`(defn ~name ~params
(let [~old-state (deref ~state)
~tmp-state (ref ~old-state)
~state-updater (fn [& args#]
(ref-set ~tmp-state (apply assoc-in-mult (deref ~tmp-state) args#)))]
(dosync ~@(map body-rewriter body))
(dosync (ref-set ~state (deref ~tmp-state)))))))
;; Application Logic
(defrecord StrokePoint [x y p])
(defrecord AppState [pen-x pen-y pen-pres strokes current-stroke])
(def the-state (ref (AppState. 0.0 0.0 0.0 (vector) (vector) )))
(defstatetrafo process-input the-state [x y p]
(let [in-stroke (not (nil? (§ :current-stroke)))
number-of-strokes (count (§ :strokes))]
(:= :pen-x x :pen-y y :pen-pres p)
(cond
(and (> p 0.0) in-stroke)
(:=
:current-stroke (conj (§ :current-stroke) (StrokePoint. x y p)))
(and (> p 0.0) (not in-stroke))
(:=
:current-stroke (vector (StrokePoint. x y p)))
(and (= p 0.0) in-stroke)
(:=
:current-stroke nil
[:strokes number-of-strokes] (§ :current-stroke)))))
;; OpenGL
;; Geometry
(defn plus [p1 p2] (StrokePoint. (+ (:x p1) (:x p2)) (+ (:y p1) (:y p2)) (:p p1)))
(defn scale [s p] (StrokePoint. (* s (:x p)) (* s (:y p)) (:p p)))
(defn minus
([p] (StrokePoint. (- (:x p)) (- (:y p)) (:p p)))
([p q] (plus p (minus q))))
(defn minusp [pair] (minus (nth pair 0) (nth pair 1)))
(defn norm [p] (clojure.contrib.math/sqrt (+ (* (:x p) (:x p)) (* (:y p) (:y p)))))
(defn normalize [p] (let [n (norm p)] (if (= n 0.0) nil (StrokePoint. (/ (:x p) n) (/ (:y p) n) (:p p)))))
(defn rot90 [p] (StrokePoint. (- (:y p)) (:x p) (:p p)))
;; Construct Triangle Strip
(defn stroke-point-to-strip-vertices [pair]
(let [point (nth pair 0) ntangent (nth pair 1)]
(if (nil? ntangent)
[]
(let [orth (rot90 ntangent)
sorth (scale (* 1.3 (:p point)) orth)
p1 (minus point sorth)
p2 (plus point sorth)]
[(:x p1) (:y p1)
(:x p2) (:y p2)]))))
(defn points-to-tangents [points]
(let [pairs (partition 2 1 points)
diffs (map minusp pairs)
tangents (map normalize diffs)]
(concat [(first tangents)] tangents)))
(defn stroke-to-coord-data [stroke]
(let [tangents (points-to-tangents stroke)
p-t-pairs (map list stroke tangents)
nested-floats (map stroke-point-to-strip-vertices p-t-pairs)]
(flatten nested-floats)))
;; OpenGL Utility Functions
(defn make-float-buffer [^GL2 gl2 target data usage]
(let [buffers (int-array 1)
float-buffer (FloatBuffer/wrap (float-array data))]
(. float-buffer rewind)
(let [count (. float-buffer remaining)
size (* count (Buffers/SIZEOF_FLOAT))]
(doto gl2
(.glGenBuffers 1 buffers 0)
(.glBindBuffer target (aget buffers 0))
(.glBufferData target size float-buffer usage))
[(aget buffers 0) count])))
;; OpenGL State
(defrecord OpenGLState [stroke-buffer])
(def opengl-state (ref (OpenGLState. (vector))))
(defstatetrafo update-opengl-state opengl-state [gl strokes]
(let [number-of-strokes (count strokes)
number-of-strokes-buffered (count (§ :stroke-buffer))]
(if (> number-of-strokes number-of-strokes-buffered)
(doseq [stroke-index (range number-of-strokes-buffered number-of-strokes)]
(let [stroke (nth strokes stroke-index)
data (stroke-to-coord-data stroke)
[buffer-id buffer-size] (make-float-buffer gl (. GL GL_ARRAY_BUFFER) data (. GL GL_STATIC_DRAW))]
(:= [:stroke-buffer stroke-index] [buffer-id (/ buffer-size 2)] ))))))
;; OpenGL Callbacks (Actual Drawing Functions)
(def gl-event-listener
(proxy [GLEventListener] []
(display [^javax.media.opengl.GLAutoDrawable drawable]
(let [^GL2 gl (.. drawable getGL getGL2)
mystate (dosync @the-state)
x (-> mystate :pen-x)
y (-> mystate :pen-y)]
(update-opengl-state gl (:strokes mystate))
(doto gl
(.glClear (. GL GL_COLOR_BUFFER_BIT))
(.glColor3f 0 0 0)
(.glBegin (. GL2 GL_QUADS))
(.glVertex3f (- x 2) (- y 2) 0)
(.glVertex3f (+ x 2) (- y 2) 0)
(.glVertex3f (+ x 2) (+ y 2) 0)
(.glVertex3f (- x 2) (+ y 2) 0)
(.glEnd))
(doseq [[buffer-id vertex-number] (:stroke-buffer @opengl-state)]
(doto gl
(.glBindBuffer (. GL GL_ARRAY_BUFFER) buffer-id)
(.glEnableClientState (. GL2 GL_VERTEX_ARRAY))
(.glVertexPointer (int 2) (int (. GL GL_FLOAT)) (int (* 2 (. Buffers SIZEOF_FLOAT))) (long 0))
(.glDrawArrays (. GL2 GL_TRIANGLE_STRIP) 0 vertex-number)
(.glDisableClientState (. GL2 GL_VERTEX_ARRAY))
(.glBindBuffer (. GL GL_ARRAY_BUFFER) 0)
))
(. gl glBegin (. GL2 GL_TRIANGLE_STRIP))
(if (:current-stroke mystate)
(doseq [[x y] (partition 2 (stroke-to-coord-data (:current-stroke mystate)))]
(. gl glVertex3f x y 0.0)))
(. gl glEnd)))
(displayChanged [drawable mode-changed device-changed])
(init [^javax.media.opengl.GLAutoDrawable drawable]
(let [gl (.. drawable getGL getGL2)]
(prn (. drawable getChosenGLCapabilities))
(doto gl
(.glClearColor 1.0 1.0 1.0 0.0)
(.glShadeModel (. GL2 GL_FLAT))
(.glEnable (. GL2 GL_MULTISAMPLE)))))
(reshape [^javax.media.opengl.GLAutoDrawable drawable x y width height]
(let [gl (.. drawable getGL getGL2)]
(doto gl
(.glMatrixMode (. GL2 GL_PROJECTION))
(.glLoadIdentity)
(.glOrtho 0 width height 0 -1 1)
(.glMatrixMode (. GL2 GL_MODELVIEW)))))))
;; Pen
(def pen-event-listener
(proxy [PenListener] []
(penLevelEvent [^PLevelEvent ev]
(let [level-to-tuple
(fn [^PLevel level]
(condp = (. level getType)
PLevel$Type/X {:x (. level value)}
PLevel$Type/Y {:y (. level value)}
PLevel$Type/PRESSURE {:p (. level value)}
{}))
initial-tuple (dosync {:x (:pen-x @the-state) :y (:pen-y @the-state) :p (:pen-pres @the-state)})
values (reduce merge initial-tuple (map level-to-tuple (. ev levels)))]
(process-input (:x values) (:y values) (:p values))))
(penButtonEvent [ev])
(penKindEvent [ev])
(penScrollEvent [ev])
(penTock [millis])))
;; Main Function
(defn go []
(let [frame (new Frame)
caps (new GLCapabilities (GLProfile/get GLProfile/GL2))
chooser (new DefaultGLCapabilitiesChooser)]
(. caps setDoubleBuffered true)
(. caps setHardwareAccelerated true)
(. caps setSampleBuffers true)
(. caps setNumSamples 4)
(let [gl-canvas (new GLCanvas caps chooser nil nil)
animator (new FPSAnimator gl-canvas 60)
pen-manager (new PenManager gl-canvas)]
(. gl-canvas addGLEventListener gl-event-listener)
(.. pen-manager pen (addListener pen-event-listener))
(. frame add gl-canvas)
(let [pixels (int-array 256)
image (. (Toolkit/getDefaultToolkit) createImage (new MemoryImageSource 16 16 pixels 0 16))
cursor (. (Toolkit/getDefaultToolkit) createCustomCursor image (new Point 0 0) "invisibleCursor")]
(. frame setCursor cursor))
(. frame setSize 300 300)
(. frame
(addWindowListener
(proxy [WindowAdapter] []
(windowClosing [event]
(. System exit 0)))))
(. animator add gl-canvas)
(. animator start)
(. frame show))))
(go)