Boot - not yet

Leiningen gets messy when trying to define complicated build paths combining Clojure, ClojureScript and possibly multiple artefacts, CSS compilations etc in one build process/configuration file.

Since Clojure 1.7.0 there can be support for multiple platforms in one codebase through cljc-files. This is awesome but must be leveraged.

Boot solves a lot of these problems by composing builds just like ordinary functions. To overcome problems with mutable file systems, boot uses an immutable file system called Filesets. Very clever.

I'm investigating wheter a semi-complex build process would benefit from moving from two repos containing a backend server and an administration client respectively.

I found out that there's no selective auto-reloading of code like in the amazing Figwheel. This limits the usability of boot for doing rich web app development at the moment. I'm sure this will change very soon.

I'll stick with Leiningen for some more, but keeps my eyes wide open for improvements in the Boot project.

Terse explanation of re-frame

Re-frame is an single page web app event-loop machinery built on Reagent/React.js.

Application state app-db is stored in one (1) reagent atom.

By using reagent's reactions, one can build up a graph, chained reactions, where more costly operations can be "shielded" behind reactions which wont report a change if the reactions view from a new app-db is the same immutable data structure as it was in the previous app-db.

The redrawing of the DOM/screen is triggered by requestAnimationFrame. Given a standard refresh rate of 60 Hz, this trigger will occur every 16.67 ms, if the program is done with it's previous recalculation by then.

Re-frame uses hiccup-style syntax feed to the Reacts Virtual DOM.

There can be several related reagent atom-constructs called reactions, containing derived state ("views") of the app-state.

To make state changes in the application, initialization code or callback functions from event-emitters like the DOM, timers, network connections should send dispatch messages. These are put in a core.async channel and handled by registered handlers.

There's two special types of dispatch, dispatch-sync which handles the event directly, and does not put it in the core.async channel. The other special dispatch is putting ^:flush-dom metadata on the dispatched event, which doesn't wait for a requestAnimationFrame event before re-rendering the reactive data structures.

Events are sent as a very simple vector format. Like a lisp sexpression the first item of the vector should be a symbol (keyword) for which function (handler) that should be handling this sexpression (event). The rest of the vector is a free-form argument vector.

Handlers are registered with (register-sub :name-of-event handler-fn)

The handler-fn has the same purpose as a clojure reduce-function, its arguments are the app-state (derefed!) and the incoming event.

The handler-fn returns the new application-state, which re-frame will reset! into app-db.

Re-frame offers a simple middleware-abstraction, where debug/logging-statements can be put. Another powerful middleware is path, that works as a lens, making the handler-fn simpler by making it work on a specific path in the app-db.

I guess one could create some middleware for simplifying the use of datascript as application state atom.

The handler should implement the logic on what should happend to an incoming event, potentially based on the app-db state.  The re-frame readme mentions finite state machines as a fruitful abstraction.

bigdec, decimal?, integer?

The function bigdec returns a java.math.BigDecimal.

If you want to test it you can use decimal? instead of defining your own function bigdec?

integer? does not test the value for not having decimals, but for being some natively decimal number type.

(defn no-decimal-part?
  "converts the number to bigdec,
  make sure a division by
  zero has remainder exactly zero"
  [number]
  (zero? (.remainder (bigdec number) java.math.BigDecimal/ONE)))

is better to use to make sure the bigdec does not have any decimals.

Limit in Datomic

It's possible to use limit (SQL-style) in Datomic when using the new Pull API. The Pull API seems to be a very ambitious API which can even make graph-walks and more.

Talk about a web application architechture

This Monday I held a talk at my company Agical presenting a sketch of an interesting web application architecture. One could call me a Cognitect fan-boy, but I'm proud of it. Before getting to the technical stuff I discussed a bit on modelling things.

I've always think modelling things are super complicated to get right, especially with ie SQL or single tables like Excel. Probably I try to take to many things into account at the same time, but hey, the world is complicated, so why shouldn't I try to model it at least a bit more like how it really is?  

The architechture, then.

Something along these lines: React/Om talking with DataScript via core.async channels and to the server. The server running Pedestal and Datomic (and whatever other data source necessary - Datomic would be the transaction engine).

It's a bit bold to claim that even database queries could be reused in the client, this is likely not the case, but the model could be similar in structure on both client and server.

The controller in the client and server would consist of a core.async pub/sub setup, although it feels a bit hard to make sure everything is setup right, it feels a bit like we are in multimethods all over again.

The start and stop of the application would be taken care of through Component. And off we go!

Extremely useful resources 
 for preparing this talk was

CatChat (a DataScript example app)

David Nolen's blog posts of both core.async and Om

Om intermediate tutorial on Datomic integration

O'Reilly's Webinar about Pedestal with Ryan Neufeld (2 h YouTube video)

Things I stumbled upon was
Trying to visualize even a simple Pedestal router setup was quite error prone. The interceptors got wrong and all over the place.

Setting up ClojureScript to compile Om. It's mostly the Leiningen project.clj options that's hard to understand. Validation of options or warnings when unused options occur would be useful.

Programming core.async channels in the nRepl often locked up the repl in strange ways, from which I had to re-require the namespaces.

Getting my head around Om and it's quite messy formatting syntax, especially when it comes to catching events from various things. Shouldn't one put small functions inside the elements maps? Maybe listeners should be defined separately from the DOM-structure.

Maybe there's even room for an abstraction over the events, because now most of it is quite OO that one has to get take back tree-element by element.

There are many fascinating papers about CRDT and similar techniques, but still it's not clear on how to structure the events from the application and the event "schema" between client and server, especially with regards to CRDT at similar. An especially delicate thing close to my heart is the ability to show where other people are editing, so edits not collide to badly.

Pub/sub "reactions" I think could be replaced with transducers.

Clojure web security

This post about Clojure web security is a must read. Not only does it summarize many things that can go bad, but also shows just how severe it can be to read data with read-string - it looks like it can execute almost any code and construct any availiable java class! Scarier than I knew. Thanks.

Steam-roller for nested Clojure data structures

There was a question on the Clojure Google group about structural sharing in rrb-vectors visavi a zipper-construction.

As we all know, it's not very easy to be certain of how much memory is allocated in Java/Clojure since the pointers are sometimes reused, and the situation gets even more complicated (for the better) when we have structural sharing and immutable objects.

The Clojure data structures are constructed from Object arrays, which creates shallow nested trees of either vector elements or key-value pairs in the case of PersistentMaps. Likely one could use reflection to dig deeper into the the inner parts of these data structures and arrays, Iroh and Clj-wallhack seems both worthy to try out.

I created a small function for getting a pointer to every  Clojure data structure from a nested structure (without taking care of it's aforementioned internals, though).

I think the best way would be to count objects with VisualVM or some other profiler, though.

Voyuerism in Datomic

Datomic is powerful. Did you know you can see everything you have ever done to your database?

(ns datomic.voyuer
  (:require [datomic.api :as d :refer [db q]]
            [clojure.pprint :refer [pprint]]))

(def uri "datomic:mem://showoff")

(def schema [{:db/id (d/tempid :db.part/db)
               :db/ident :person/name
               :db/valueType :db.type/string
               :db/cardinality :db.cardinality/one
               :db/doc "A persons name"
               :db.install/_attribute :db.part/db}
              {:db/id (d/tempid :db.part/db)
               :db/ident :person/email
               :db/unique :db.unique/identity
               :db/valueType :db.type/string
               :db/cardinality :db.cardinality/one
               :db/doc "A persons email, must be unique"
               :db.install/_attribute :db.part/db}])


(d/create-database uri)
(def conn (d/connect uri))
(d/transact conn schema)

(d/transact conn [{:db/id (d/tempid :db.part/user)
                   :person/name "Linus"
                   :person/email "itisnot@linusericsson.se"}])

(pprint (seq (d/datoms (db conn) :eavt)))

And the result is a sequence of Datoms which can be seen as entities.

These datoms can be decoded as

[Entity id , Attribute (as a pointer to the attributes entity id), Value (as a clojure literal), Tx (as a pointer to the tx-entity), boolean added?].

I did just add empty lines between the datom-numbers and some titles where apropiate. Everything from the system partition the added schema and data is in the list. It's even well documented!

(#datom[0 10 :db.part/db 13194139533312 true] ;; datom 0 is the System partition
 #datom[0 11 0 13194139533366 true] ;;13194139533366 a txInstant @ beginning of epoch
 #datom[0 11 3 13194139533366 true]
 #datom[0 11 4 13194139533366 true]
 #datom[0 12 20 13194139533366 true]
 #datom[0 12 21 13194139533366 true]
 #datom[0 12 22 13194139533366 true]
 #datom[0 12 23 13194139533366 true]
 #datom[0 12 24 13194139533366 true]
 #datom[0 12 25 13194139533366 true]
 #datom[0 12 26 13194139533366 true]
 #datom[0 12 27 13194139533366 true]
 #datom[0 12 56 13194139533368 true]
 #datom[0 12 57 13194139533368 true]
 #datom[0 12 58 13194139533368 true]
 #datom[0 12 59 13194139533368 true]
 #datom[0 12 60 13194139533368 true]
 #datom[0 12 61 13194139533368 true]
 #datom[0 13 10 13194139533366 true]
 #datom[0 13 11 13194139533366 true]
 #datom[0 13 12 13194139533366 true]
 #datom[0 13 13 13194139533366 true]
 #datom[0 13 14 13194139533366 true]
 #datom[0 13 15 13194139533366 true]
 #datom[0 13 16 13194139533366 true]
 #datom[0 13 17 13194139533366 true]
 #datom[0 13 18 13194139533366 true]
 #datom[0 13 19 13194139533366 true]
 #datom[0 13 39 13194139533366 true]
 #datom[0 13 40 13194139533366 true]
 #datom[0 13 41 13194139533366 true]
 #datom[0 13 42 13194139533366 true]
 #datom[0 13 43 13194139533366 true]
 #datom[0 13 44 13194139533366 true]
 #datom[0 13 45 13194139533366 true]
 #datom[0 13 46 13194139533366 true]
 #datom[0 13 47 13194139533366 true]
 #datom[0 13 50 13194139533366 true]
 #datom[0 13 51 13194139533366 true]
 #datom[0 13 52 13194139533366 true]
 #datom[0 13 62 13194139533368 true]
 #datom[0 13 63 13194139534312 true]
 #datom[0 13 64 13194139534312 true]
 #datom[0 14 54 13194139533366 true]
 #datom[0 14 55 13194139533366 true]
 #datom[0 62 "Name of the system partition. The system partition includes the core of datomic, as well as user schemas: type definitions, attribute definitions, partition definitions, and data function definitions." 13194139533375 true]


 #datom[1 10 :db/add 13194139533312 true]
 #datom[1 62 "Primitive assertion. All transactions eventually reduce to a collection of primitive assertions
 and retractions of facts, e.g. [:db/add fred :age 42]." 13194139533375 true]

 #datom[2 10 :db/retract 13194139533312 true]
 #datom[2 62 "Primitive retraction. All transactions eventually reduce to a collection of assertions and retractions of facts, e.g. [:db/retract fred :age 42]." 13194139533375 true]

 #datom[3 10 :db.part/tx 13194139533312 true]
 #datom[3 62 "Partition used to store data about transactions. Transaction data always includes a :db/txInstant which is the transaction's timestamp, and can be extended to store other information at transaction granularity." 13194139533375 true]

 #datom[4 10 :db.part/user 13194139533312 true]
 #datom[4 62 "Name of the user partition. The user partition is analogous to the default namespace in a programming language, and should be used as a temporary home for data during interactive development." 13194139533375 true]

 #datom[10 10 :db/ident 13194139533312 true]
 #datom[10 40 21 13194139533366 true]
 #datom[10 41 35 13194139533366 true]
 #datom[10 42 38 13194139533366 true]
 #datom[10 62 "Attribute used to uniquely name an entity." 13194139533375 true]

 #datom[11 10 :db.install/partition 13194139533312 true]
 #datom[11 40 20 13194139533366 true]
 #datom[11 41 36 13194139533366 true]
 #datom[11 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a partition." 13194139533375 true]

 #datom[12 10 :db.install/valueType 13194139533312 true]
 #datom[12 40 20 13194139533366 true]
 #datom[12 41 36 13194139533366 true]
 #datom[12 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a value type." 13194139533375 true]

 #datom[13 10 :db.install/attribute 13194139533312 true]
 #datom[13 40 20 13194139533366 true]
 #datom[13 41 36 13194139533366 true]
 #datom[13 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as an attribute." 13194139533375 true]

 #datom[14 10 :db.install/function 13194139533312 true]
 #datom[14 40 20 13194139533366 true]
 #datom[14 41 36 13194139533366 true]
 #datom[14 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a data function." 13194139533375 true]

 #datom[15 10 :db/excise 13194139533312 true]
 #datom[15 40 20 13194139533366 true]
 #datom[15 41 35 13194139533366 true]

 #datom[16 10 :db.excise/attrs 13194139533312 true]
 #datom[16 40 20 13194139533366 true]
 #datom[16 41 36 13194139533366 true]

 #datom[17 10 :db.excise/beforeT 13194139533312 true]
 #datom[17 40 22 13194139533366 true]
 #datom[17 41 35 13194139533366 true]

 #datom[18 10 :db.excise/before 13194139533312 true]
 #datom[18 40 25 13194139533366 true]
 #datom[18 41 35 13194139533366 true]

 #datom[19 10 :db.alter/attribute 13194139533312 true]
 #datom[19 40 20 13194139533366 true]
 #datom[19 41 36 13194139533366 true]
 #datom[19 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will alter the definition o
f existing attribute v." 13194139533375 true]

 #datom[20 10 :db.type/ref 13194139533312 true]
 #datom[20 39 :ref 13194139533366 true]
 #datom[20 62 "Value type for references. All references from one entity to another are through attributes with this value type." 13194139533375 true]

 #datom[21 10 :db.type/keyword 13194139533312 true]
 #datom[21 39 :key 13194139533366 true]
 #datom[21 62 "Value type for keywords. Keywords are used as names, and are interned for efficiency. Keywords map to the native interned-name type in languages that support them." 13194139533375 true]

 #datom[22 10 :db.type/long 13194139533312 true]
 #datom[22 39 :int 13194139533366 true]
 #datom[22 62 "Fixed integer value type. Same semantics as a Java long: 64 bits wide, two's complement binary representation." 13194139533375 true]

 #datom[23 10 :db.type/string 13194139533312 true]
 #datom[23 39 :string 13194139533366 true]
 #datom[23 62 "Value type for strings." 13194139533375 true]

 #datom[24 10 :db.type/boolean 13194139533312 true]
 #datom[24 39 :bool 13194139533366 true]
 #datom[24 62 "Boolean value type." 13194139533375 true]

 #datom[25 10 :db.type/instant 13194139533312 true]
 #datom[25 39 :inst 13194139533366 true]
 #datom[25 62 "Value type for instants in time. Stored internally as a number of milliseconds since midnight, January 1, 1970 UTC. Representation type will vary depending on the language you are using." 13194139533375 true]

 #datom[26 10 :db.type/fn 13194139533312 true]
 #datom[26 39 :datomic/fn 13194139533366 true]
 #datom[26 62 "Value type for database functions. See Javadoc for Peer.function." 13194139533375 true]

 #datom[27 10 :db.type/bytes 13194139533312 true]
 #datom[27 39 :bytes 13194139533366 true]
 #datom[27 62 "Value type for small binaries. Maps to byte array on the JVM." 13194139533375 true]

 #datom[35 10 :db.cardinality/one 13194139533312 true]
 #datom[35 62 "One of two legal values for the :db/cardinality attribute. Specify :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued a
ttributes." 13194139533375 true]

 #datom[36 10 :db.cardinality/many 13194139533312 true]
 #datom[36 62 "One of two legal values for the :db/cardinality attribute. Specify :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes." 13194139533375 true]

 #datom[37 10 :db.unique/value 13194139533312 true]
 #datom[37 62 "Specifies that an attribute's value is unique. Attempts to create a new entity with a colliding value for a :db.unique/value will fail." 13194139533375 true]

 #datom[38 10 :db.unique/identity 13194139533312 true]
 #datom[38 62 "Specifies that an attribute's value is unique. Attempts to create a new entity with a colliding value for a :db.unique/value will become upserts." 13194139533375 true]

 #datom[39 10 :fressian/tag 13194139533312 true]
 #datom[39 40 21 13194139533366 true]
 #datom[39 41 35 13194139533366 true]
 #datom[39 44 true 13194139533366 true]
 #datom[39 62 "Keyword-valued attribute of a value type that specifies the underlying fressian type
used for serialization." 13194139533375 true]

 #datom[40 10 :db/valueType 13194139533312 true]
 #datom[40 40 20 13194139533366 true]
 #datom[40 41 35 13194139533366 true]
 #datom[40 62 "Property of an attribute that specifies the attribute's value type. Built-in value types include, :db.type/keyword, :db.type/string, :db.type/ref, :db.type/instant, :db.type/long, :db.type/bigdec, :db.type/boolean, :db.type/float, :db.type/uuid, :db.type/double, :db.type/bigint,  :db.type/uri." 13194139533375 true]

 #datom[41 10 :db/cardinality 13194139533312 true]
 #datom[41 40 20 13194139533366 true]
 #datom[41 41 35 13194139533366 true]
 #datom[41 62 "Property of an attribute. Two possible values: :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes. Defaults to :db.cardinality/one." 13194139533375 true]

 #datom[42 10 :db/unique 13194139533312 true]
 #datom[42 40 20 13194139533366 true]
 #datom[42 41 35 13194139533366 true]
 #datom[42 62 "Property of an attribute. If value is
 :db.unique/value, then attribute value is unique to each entity. Attempts to insert a duplicate value for a temporary entity id will fail. If value is :db.unique/identity, then attribute value is unique, and upsert is enabled. Attempting to insert a duplicate value for a temporary entity id will cause all attributes associated with that temporary id to be merged with the entity already in the database. Defaults to nil." 13194139533375 true]

 #datom[43 10 :db/isComponent 13194139533312 true]
 #datom[43 40 24 13194139533366 true]
 #datom[43 41 35 13194139533366 true]
 #datom[43 62 "Property of attribute whose vtype is :db.type/ref. If true, then the attribute is a component of the entity referencing it. When you query for an entire entity, components are fetched automatically. Defaults to nil." 13194139533375 true]

 #datom[44 10 :db/index 13194139533312 true]
 #datom[44 40 24 13194139533366 true]
 #datom[44 41 35 13194139533366 true]
 #datom[44 62 "Property of an attribute. If true, create an AVET index for th
e attribute. Defaults to false." 13194139533375 true]

 #datom[45 10 :db/noHistory 13194139533312 true]
 #datom[45 40 24 13194139533366 true]
 #datom[45 41 35 13194139533366 true]
 #datom[45 62 "Property of an attribute. If true, past values of the attribute are not retained after indexing. Defaults to false." 13194139533375 true]

 #datom[46 10 :db/lang 13194139533312 true]
 #datom[46 40 20 13194139533366 true]
 #datom[46 41 35 13194139533366 true]
 #datom[46 62 "Attribute of a data function. Value is a keyword naming the implementation language of the function. Legal values are :db.lang/java and :db.lang/clojure" 13194139533375 true]

 #datom[47 10 :db/code 13194139533312 true]
 #datom[47 40 23 13194139533366 true]
 #datom[47 41 35 13194139533366 true]
 #datom[47 51 true 13194139533366 true]
 #datom[47 62 "String-valued attribute of a data function that contains the function's source code." 13194139533375 true]

 #datom[48 10 :db.lang/clojure 13194139533312 true]
 #datom[48 62 "Value of :db/lang attribute, spec
ifying that a data function is implemented in Clojure." 13194139533375 true]

 #datom[49 10 :db.lang/java 13194139533312 true]
 #datom[49 62 "Value of :db/lang attribute, specifying that a data function is implemented in Java." 13194139533375 true]

 #datom[50 10 :db/txInstant 13194139533312 true]
 #datom[50 40 25 13194139533366 true]
 #datom[50 41 35 13194139533366 true]
 #datom[50 44 true 13194139533366 true]
 #datom[50 62 "Attribute whose value is a :db.type/instant. A :db/txInstant is recorded automatically with every transaction." 13194139533375 true]

 #datom[51 10 :db/fulltext 13194139533312 true]
 #datom[51 40 24 13194139533366 true]
 #datom[51 41 35 13194139533366 true]
 #datom[51 62 "Property of an attribute. If true, create a fulltext search index for the attribute. Defaults to false." 13194139533375 true]

 #datom[52 10 :db/fn 13194139533312 true]
 #datom[52 40 26 13194139533366 true]
 #datom[52 41 35 13194139533366 true]
 #datom[52 62 "A function-valued attribute for direct use by transactions and qu
eries." 13194139533375 true]

 #datom[53 10 :db.bootstrap/part 13194139533312 true]

 #datom[54 10 :db.fn/retractEntity 13194139533366 true]
 #datom[54 46 48 13194139533366 true]
 #datom[54 47 "(clojure.core/fn [db e] (datomic.builtins/build-retract-args db e))" 13194139533366 true]
 #datom[54 62 "Retract all facts about an entity, including references from other entities and component attributes recursively." 13194139533375 true]

 #datom[55 10 :db.fn/cas 13194139533366 true]
 #datom[55 46 48 13194139533366 true]
 #datom[55 47 "(clojure.core/fn [db e a ov nv] (datomic.builtins/compare-and-swap db e a ov nv))" 13194139533366 true]
 #datom[55 62 "Compare and swap the value of an entity's attribute." 13194139533375 true]

Database value types: 

#datom[56 10 :db.type/uuid 13194139533368 true]
 #datom[56 39 :uuid 13194139533368 true]
 #datom[56 62 "Value type for UUIDs. Maps to java.util.UUID on the JVM." 13194139533375 true]

 #datom[57 10 :db.type/double 13194139533368 true]
 #datom[57 39 :double 13194139533368 true]
 #datom[57 62 "Float
ing point value type. Same semantics as a Java double: double-precision 64-bit IEEE 754 floating point." 13194139533375 true]

 #datom[58 10 :db.type/float 13194139533368 true]
 #datom[58 39 :float 13194139533368 true]
 #datom[58 62 "Floating point value type. Same semantics as a Java float: single-precision 32-bit IEEE 754 floating point." 13194139533375 true]

 #datom[59 10 :db.type/uri 13194139533368 true]
 #datom[59 39 :uri 13194139533368 true]
 #datom[59 62 "Value type for URIs. Maps to java.net.URI on the JVM." 13194139533375 true]

 #datom[60 10 :db.type/bigint 13194139533368 true]
 #datom[60 39 :bigint 13194139533368 true]
 #datom[60 62 "Value type for arbitrary precision integers. Maps to java.math.BigInteger on the JVM." 13194139533375 true]

 #datom[61 10 :db.type/bigdec 13194139533368 true]
 #datom[61 39 :bigdec 13194139533368 true]
 #datom[61 62 "Value type for arbitrary precision floating point numbers. Maps to java.math.BigDecimal on the JVM." 13194139533375 true]

 #datom[62 10 :db/doc 13194139533368 true]
 #datom[62 40 23 13194139533368 true]
 #datom[62 41 35 13194139533368 true]
 #datom[62 51 true 13194139533368 true]
 #datom[62 62 "Documentation string for an entity." 13194139533375 true]

The added schema:

 #datom[63 10 :person/name 13194139534312 true]
 #datom[63 40 23 13194139534312 true]
 #datom[63 41 35 13194139534312 true]
 #datom[63 62 "A persons name" 13194139534312 true]

 #datom[64 10 :person/email 13194139534312 true]
 #datom[64 40 23 13194139534312 true]
 #datom[64 41 35 13194139534312 true]
 #datom[64 42 38 13194139534312 true]
 #datom[64 62 "A persons email, must be unique" 13194139534312 true]

TxInstants, starts at entity id 13194139533366:

 #datom[13194139533366 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533366 true]
 #datom[13194139533368 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533368 true]
 #datom[13194139533375 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533375 true]
 #datom[13194139534312 50 #inst "2014-07-17T16:30:23.535-00:00" 13194139534312 true]
 #datom[13194139534313 50 #inst "2014-07-17T16:30:23.541-00:00" 13194139534313 true]

Added facts

 #datom[17592186045418 63 "Linus" 13194139534313 true]
 #datom[17592186045418 64 "itisnot@linusericsson.se" 13194139534313 true])

Spy with Timbre

Peter Taoussanis Timbre logging library is awesome. It's dead easy to get started with - add a dependecy and you're done.

One really usable features in it (apart from the ordinary "write this to log" functions) is spy.

spy let's you wrap an expression and log both the expression and it's result, like so:

(require '[taoensso.timbre :as timbre])

(defn some-function [argh]
    (timbre/spy (reverse argh)))

(defn some-other-function [x]
    (timbre/spy (map rand-int x)))

(some-other-function (some-function [1 2 34]))
2014-Jul-17 18:12:06 +0200 Albatron-5 DEBUG [user] - (reverse argh) (34 2 1)
2014-Jul-17 18:12:06 +0200 Albatron-5 DEBUG [user] - (map rand-int x) (30 0 0)
-> (30 0 0)
 

The result of the two above functions are not apparent, and there is good reason to keep an eye on intermediate results. This can be tedious when not in light-table. Not so with spy.

spy function documentation

The risks with TDD

Lately there has been a lot of debate about Test Driven Development. For me I started consider the usage of TDD when Rich Hickey formulated it as "you wouldn't drive a car by crashing into the guard-rails". I would change the metaphor slightly to mean "you don't design cars not crash into the guard-rails".

Whether you hit the guard-rails or not is not a decent measure whether a car is good or not. The higher idea of a car is not something that don't crashes into the guard rails. The higher idea of a car is more of something like a tool to transport yourself and some more people and stuff quickly, safe, cheap, and in a joyful manner from where you are now to some place farther away than you could comfortably walk. No guard rails involved.

"But the test-cases is a form of strict requirement specifications!" you say. Of course they are. It's necessary to specify requirements in some form, to detect logic inconsistencies and communicate the design (to make others able to criticize it). Write test cases is a way of doing this, but usually not the best way.

Another problem which is perpendicular to get a logically consistent view of the problem is the problem of avoiding/catch trivial implementation mistakes early. By trivial I mean something like a comparison with > instead of >=. These mistakes usually don't really touch the higher idea of the program, but are of course crucial to get correct. Handmade test cases, asserts, contract programming and even generative testing are great tools for this. Don't mix solving these bugs with detecting higher level logic inconsistencies of your code. Trivial bugs are relieving to correct, but never of any significant value for the program. It's not the hard problem.

My baby message-queue example
The risk with test-driven development is that you start to code before you know what to code. It is of course a good way to get yourself out of analysis paralysis, but in my experience, there are to little 'analysis paralysis' compared to 'cowboy coding that gets impossible to manage and extend later on'.

Let's say you are in need of some kind of messaging system. Your system is spread over several computers, and you have already utilized ad-hoc socket stream formats and various REST-APIs in an unsustainable way, just to get the whole thing going. You shrug at the idea of expanding the system. Wouldn't it be nice to solve this problem once and for all?

Let's say we try to formulate some test cases about how this messaging system could work. Obviously you'll need some way to connect the program to the messaging system bus, and some way to send and receive messages.

It could be a test-case like this:

(let [connection (connect "some-message-bus")]
   (is (up? connection)))

and

(let [connection (connect "some-message-bus")]
   (send! connection "hello")
   (is (== (receive! connection) "hello"))))

This is great! It is simple! I can see that there's a function connect that need to be implemented. This needs to return an object that, given to a function up? returns true.

I also want to be able to send and receive things on each bus.

This is actually highly enlightening. It's a minimalistic interface but it captures a small API. I know what I want and we can even discuss parts of the design given these small lines of code.

Of course there's a ton of various things I need to add - some event loop facility, networking support to be able to connect remote computers, reliance of the message queue, potentially other behaviours, but I could very well code up something that would work quite OK in process just given this simple test. Almost like magic!

Never underestimate the real problem in distributed computing
It could easily be the case that we actually would be in need of quite a different solution than the one we thought we needed - maybe we didn't parallelize the problem enough and did all the socket shuffling in vain (I've seen that more than once). And how do we handle that connections are lost? Can we unsubscribe a bus? What happens if we subscribe to many queues. What if servers are down? New servers join the cluster? Authentication? Am I the first person in the Universe facing this particular problem and is it unique of its kind? (that last one was a rhetoric question).

Depending on the specific problem the computing cluster should solve, there are many different ways to approach it. My small TDD approach has the insidious side-effect that it actually makes me narrow down my view of the problem way too early. It's very hard to kill code you wrote start over, almost from scratch. Good test takes effort to specify (and rightly so). To refactor tests probably takes even more effort than to refactor code.

The TDD way to navigate through the large space of solutions, can easily get stuck on a local maxima.

I can do nothing but approve Rich Hickeys idea of "hammock time". You really have to understand and be able to keep the whole problem you try to solve, and as many of its parameters and quirks. One good way to do this is really to try to solve the problem (preferably on paper) and after that see how others did solve similar problems, code up a small prototype, see how others code would solve the same problem, make sure you know how to the whole stack would work. When this work is done: TDD would work just fine.

TDD is a great scaffold, but a most often a really shitty sketch. Actually test cases have too high fidelity and is to slow to write and change, and you can mistake them for "real test cases that should be used to test the final solution on".

Test cases is not a good blue print either - test cases does not say how the solution should work, only how it should behave. If you really want to code this way (which is really powerful), use a declarative programming language, like datalog instead of abusing your precious brain time with being a manual  compiler for your own, made up, non-standard, most likely hard-to-understand declarative programming language. (Those are more common than you might think).

In summary
The major risk with TDD is that one get carried away on the wrong track, starts to solve some problem, get some kick from passing tests, continues in that direction for more passing tests kicks, and never really takes the opportunity to really think and reason systematically about the whole problem one have at hand.

Don't be mean to yourself and your friends, always think hard on your problem and possible solutions AFK before coding. If that is not possible, something is wrong, and your code will likely be as messy as the problem it tries to solve.

When you know exactly what the outlines of the problem are and how you want to solve this problem - then TDD is one of several tools to get your code super duper great. Use it accordingly. Thanks.

∑ 2013

This year has been a very enlightening one. I have peeled the onion and learnt that many things earlier considered magic, was not very magic at all, yet elegant and far from trivial to implement myself. Hopefully I'm less prone to get stuck in trying to reimplement these features in the future.

I have finally learnt to see through all the abstract classes of java, finally understood the benefits interface and how clojure generates bytecode. Maybe a bit late, but I'm always learn things backwards anyway.

I've been reading some really good books and articles. Among them are

Brian Goetz - Java Concurrency in practice in which I learnt much more about the volatile and various atomic constructs in java.

Fred Hébert - Learn You Some Erlang for great good! - apart from the marvelous illustrations it's fun to see what Erlangs strengths really are, among them a very efficient implementation of green threads and the selective message receiving, removing much complexity in parsing-like functionality.

The JVM serialization benchmarking results - told me about the existence of Avro and Kryo, among others. I later found out about Kryonet, which I hope to try out further. I also read up on Fressian,

Pedestal.io entered my life. It will be very hard to start develop web applications in other frameworks after trying out this beast.

I re-read The Joy of Clojure (Fogus, Chouser) and realized I had missed most of it at the first read. The talk from Hugo Duncan on Debuging in Clojure helped me grasp the mind-blowing feature of starting a repl in the middle of an error.

Entered Garbage Collections Eden area when I visited a Lisp meet up in Gothenburg. People discussed to implement their own Garbage Collector and I was thinking "Impossible!". Afterwards I read up on the subject. It's not impossible at all, and it made me a somewhat better programmer to know it's not magic and I even dare to think I can play a bit better with garbage collecting now.

Professionally I've been able to juggle matrixes several gigabytes large in memory, which made my $500 laptop be as performant as a half rack of Proliant servers. Fun, and more than a bit scary.

I finally read through all of the Clojure source code. Much to say, yet little. The array-map (as well as the large hash-map) is likely much more conservative on allocating objects than Java Collections own implementations.

I learnt that TCP is quite a shitty protocol for throughput because of its strict ordering ACK mechanism. This explains why Storm and a lot of other applications chose to use UDP and implement their own ACK-ing mechanisms. I also learned that the SCTP protocol is quite cool, and that even Java already supports it.

I thought long and hard about compilation and compilers. I read parts of Urban Boquists PhD thesis on Code Optimization for Lazy Functional Languages, and found realized that inlining and register optimization share some similarities, although explicit register optimization likely will produce better results faster.

I wanted to find out if JIT supports the SSE and other late x86 instruction sets, and turns out it does, although it's hard to know exactly when. There's the option PrintAssembly for knowing exactly what the JIT spits out, which I hope to investigate more.

CUDA and OpenCL was getting visits from Clojure-generated guests like Sleipnir, I'm still looking for a suitable problem to squeeze into GPUs.

I also read up on Postgres SQL internals and indexing, the bitmap indexes are a cool thing. The clojure.lang.PersistentHashMap is implemented in very similar way. Could this be used to optimize the clojure.set/union and other set-operations somehow?

I finally discovered the ThreadLocal classes in java, which are potentially great for thread-bound processing, like CRC calculations or cryptographic state.

Thanks to David Nolen for continously tweeting things about relational programming I didn't even know existed.

Zach Tellman published Narrator, which is yet another stream processing library, full of clever close-to-JVM goodies.

(inc 2013)
Hopefully next year will be the year I visit some Clojure conference. I'm thinking a lot on state machines and network programming in my current job, but also visualizations, so I really hope I will be able to publish something slightly valuable regarding these issues.

Suprisingly expensive volatiles

The java keyword volatile defines a variable to always be written to main memory before some other method access it. It's important to notice that this write to main memory takes about 1000 clockcycles on a modern CPU, since the variable has to traverse three layers of cache to get there. The use of volatile should be very carefully investigated. The blog Mechanical Sympathy wrote more about the use of volatile variables two and a half year ago.

Need for counterspeed

I just had to try out the speed of different counters. Turns out clojure atoms is competitive for all but the most specialized applications.

View 2d-arrays in Incanter

There were a question the in Incanter google-group for a way to show the content of 2d-arrays as heatmaps.

I suggested him to use a function that takes the integer from the given coordinates and be careful about the axis-scaling. Trying it out my self it turned out to be a terrible suggestion. Sorry.

I looked into the source-code of incanter.chart/heat-map* and highjacked the place where the function was called and replaced that with the previously developed matrix-lookup. Simple and ugly.

Touché

Great blog post: The Perils of Future-Coding. Of course I can recognize myself in that one.

Well, the cures are: dumb down things! A lot. I will publish three examples of dumbing down things here in rapid order. See ya!

Soft Realtime Java

Anyone visiting my home could verify that garbage collection is something best left to some other entity than me. One severe problem with having a computer program looking what your program is doing trying to clean up after it, is that it sometimes gets so confused it has to trigger some divine intervention, stop the time, go through much of the allocated memory and reallocate parts not in use anymore.

This seasonally cleaning can take a while. Usually that's ok, but for more and more applications the world can't wait. One of these are when the computer is busy generating cool synth sounds for a dancefloor in real time.

To solve this issue many programs simply don't use Java, but C++ or something else where you have more control over how your memory is allocated and deallocated. This leads to nicely working software, like the Ableton Live, but also rather much work for the programmers behind it. Ok, Javas somewhat crimped primitives and the JVMs limitied abilities to reach new efficient processor instructions wouldn't make much to help it out either, but anyway.

One new player in the game is IBM's Metronome GC, that can guarantee a maximum pause time for Java GC that is about 2 ms. Of course, someone made a sample-player synth written in java with rather competitive latencies.

Garbage Collection visualization

I really like the old disk defragmenter program in Windows. The small blocks where moved all around and I could watch it for hours (well, minutes at least).


The JVM Garbage Collection is something of a black-box which sometimes just makes everything go to halt. To get a better understanding of how it is struggling with my code I wanted something like the disk defragmenter, and, Eureka! Netflix has made their own tool, gc.viz (blogpost), and also logs all the garbage collection closely in their production systems. Interesting indeed!

Leaderboard, some approaches

I'm looking into the leader board problem. In short, the problem can be expressed as the most efficient way to take the n best out of a large (often semi ordered) set with the size m (the cardinality of the total league).

The most naive version is to keep a list of the n≤m best players seen so far. If we assume no sorting among these n players, the complexity of the algorithm will be n*m, which is often unacceptable, since both m and n can be huge.

Cache the worst one the list
A slightly less naive way is to cache the least good player in the current toplist. This will, at best, give only slightly above m comparisions, given that the best players came first. Beware though, if the set is given in reversed order, the complexity once again soars to n*m.

Keep a binary tree heap
A heap is an often excellent way to select items out of a set. The most general selection, the heap-sort has a well known n log n complexity. The heap could be a candidate because it keeps the players on the top list sorted and could be a good candidate. This is not the case. The binary tree has the sad property that it can be very unbalanced and we are back again at n*m.

Keep a balanced binary tree heap!
Now we're talking! There are several trees that balance themselves in a well-known worst case time available. Actually, there are some specialized heap data structures availiable, a list from Wikipedia on Heap data structures gives the following

• 2-3 heap
• Beap 
• Binary heap
• Binomial heap
• Brodal queue
• d-ary heap
• Fibonacci heap
• Leftist heap
• Pairing heap
• Skew heap
• Soft heap
• Weak heap
• Leaf heap
• Radix heap
• Randomized meldable heap

where the Fibonacci and Brodal heaps seems to have among the best over-all performance (edit: no, the different heaps do of course have different properties depending on the insert/delete proportions). 

I'll give a try to implement the Fibonacci heap. And yes, it's one availiable in clojure already, courtesy of Mary Rose Cook, maryrosecook/fibonacciheap, built with zippers, nicely described a great blogpost. Super elegant.

Small app for finding BPMs by key-tap

With Giorgio-voice: When I was young, 15-16 I definetly wanted to become a DJ and producer. I left that for my friends in Skillnicker instead. I took the engineering route instead. And here we are.

When you want to mix two songs togheter they needed to be in the same tempo, therefore we marked our records with the number of BPMs a certain song had. We calculated in the beginning by measuring the length of 16 beats with a stop watch and look into a table. After a while there were programmes where one just could tap the space bar and get a decent result. Here is such a program in Clojure.

  The intresting part of the program is the minimal state, it's just the last time we got a key press in the frame. I will add a Kalman-filter to it soon, giving it a state of a hefty two or three variables.

Core.async for simple user-interaction

Alex Miller gives an excellent example on how to make rock-paper-scissors with core.async which made me want to try it on my mekanikmaskin - a computer-aided learning and examinating tool.

The cool thing about core.async is that it makes sort-of a reactive programming possible. Put something in a channel, and watch the program take care of it. Almost like REPLs waiting inside functions wherever you like.


And yes, it's a terrible mixture between printlns and the async constructs, but the way to create a more advanced interaction is for another day.