söndag 5 april 2015

Terse explanation of re-frame

Re-frame is an single page web app event-loop machinery built on Reagent/React.js.

Application state app-db is stored in one (1) reagent atom.

By using reagent's reactions, one can build up a graph, chained reactions, where more costly operations can be "shielded" behind reactions which wont report a change if the reactions view from a new app-db is the same immutable data structure as it was in the previous app-db.

The redrawing of the DOM/screen is triggered by requestAnimationFrame. Given a standard refresh rate of 60 Hz, this trigger will occur every 16.67 ms, if the program is done with it's previous recalculation by then.

Re-frame uses hiccup-style syntax feed to the Reacts Virtual DOM.

There can be several related reagent atom-constructs called reactions, containing derived state ("views") of the app-state.

To make state changes in the application, initialization code or callback functions from event-emitters like the DOM, timers, network connections should send dispatch messages. These are put in a core.async channel and handled by registered handlers.

There's two special types of dispatch, dispatch-sync which handles the event directly, and does not put it in the core.async channel. The other special dispatch is putting ^:flush-dom metadata on the dispatched event, which doesn't wait for a requestAnimationFrame event before re-rendering the reactive data structures.

Events are sent as a very simple vector format. Like a lisp sexpression the first item of the vector should be a symbol (keyword) for which function (handler) that should be handling this sexpression (event). The rest of the vector is a free-form argument vector.

Handlers are registered with (register-sub :name-of-event handler-fn)

The handler-fn has the same purpose as a clojure reduce-function, its arguments are the app-state (derefed!) and the incoming event.

The handler-fn returns the new application-state, which re-frame will reset! into app-db.

Re-frame offers a simple middleware-abstraction, where debug/logging-statements can be put. Another powerful middleware is path, that works as a lens, making the handler-fn simpler by making it work on a specific path in the app-db.

I guess one could create some middleware for simplifying the use of datascript as application state atom.

The handler should implement the logic on what should happend to an incoming event, potentially based on the app-db state.  The re-frame readme mentions finite state machines as a fruitful abstraction.

söndag 29 mars 2015

bigdec, decimal?, integer?

The function bigdec returns a java.math.BigDecimal.

If you want to test it you can use decimal? instead of defining your own function bigdec?

integer? does not test the value for not having decimals, but for being some natively decimal number type.

(defn no-decimal-part?
  "converts the number to bigdec,
  make sure a division by
  zero has remainder exactly zero"
  [number]
  (zero? (.remainder (bigdec number) java.math.BigDecimal/ONE)))


is better to use to make sure the bigdec does not have any decimals.

måndag 29 december 2014

Limit in Datomic

It's possible to use limit (SQL-style) in Datomic when using the new Pull API. The Pull API seems to be a very ambitious API which can even make graph-walks and more.

torsdag 23 oktober 2014

Talk about a web application architechture

This Monday I held a talk at my company Agical presenting a sketch of an interesting web application architecture. One could call me a Cognitect fan-boy, but I'm proud of it. Before getting to the technical stuff I discussed a bit on modelling things.

I've always think modelling things are super complicated to get right, especially with ie SQL or single tables like Excel. Probably I try to take to many things into account at the same time, but hey, the world is complicated, so why shouldn't I try to model it at least a bit more like how it really is?  

The architechture, then.

Something along these lines: React/Om talking with DataScript via core.async channels and to the server. The server running Pedestal and Datomic (and whatever other data source necessary - Datomic would be the transaction engine).

It's a bit bold to claim that even database queries could be reused in the client, this is likely not the case, but the model could be similar in structure on both client and server.

The controller in the client and server would consist of a core.async pub/sub setup, although it feels a bit hard to make sure everything is setup right, it feels a bit like we are in multimethods all over again.

The start and stop of the application would be taken care of through Component. And off we go!

Extremely useful resources 
 for preparing this talk was

CatChat (a DataScript example app)

David Nolen's blog posts of both core.async and Om

Om intermediate tutorial on Datomic integration

O'Reilly's Webinar about Pedestal with Ryan Neufeld (2 h YouTube video)

Things I stumbled upon was
Trying to visualize even a simple Pedestal router setup was quite error prone. The interceptors got wrong and all over the place.

Setting up ClojureScript to compile Om. It's mostly the Leiningen project.clj options that's hard to understand. Validation of options or warnings when unused options occur would be useful.

Programming core.async channels in the nRepl often locked up the repl in strange ways, from which I had to re-require the namespaces.

Getting my head around Om and it's quite messy formatting syntax, especially when it comes to catching events from various things. Shouldn't one put small functions inside the elements maps? Maybe listeners should be defined separately from the DOM-structure.

Maybe there's even room for an abstraction over the events, because now most of it is quite OO that one has to get take back tree-element by element.

There are many fascinating papers about CRDT and similar techniques, but still it's not clear on how to structure the events from the application and the event "schema" between client and server, especially with regards to CRDT at similar. An especially delicate thing close to my heart is the ability to show where other people are editing, so edits not collide to badly.

Pub/sub "reactions" I think could be replaced with transducers.

måndag 6 oktober 2014

Clojure web security

This post about Clojure web security is a must read. Not only does it summarize many things that can go bad, but also shows just how severe it can be to read data with read-string - it looks like it can execute almost any code and construct any availiable java class! Scarier than I knew. Thanks.

tisdag 12 augusti 2014

Steam-roller for nested Clojure data structures

There was a question on the Clojure Google group about structural sharing in rrb-vectors visavi a zipper-construction.

As we all know, it's not very easy to be certain of how much memory is allocated in Java/Clojure since the pointers are sometimes reused, and the situation gets even more complicated (for the better) when we have structural sharing and immutable objects.

The Clojure data structures are constructed from Object arrays, which creates shallow nested trees of either vector elements or key-value pairs in the case of PersistentMaps. Likely one could use reflection to dig deeper into the the inner parts of these data structures and arrays, Iroh and Clj-wallhack seems both worthy to try out.

I created a small function for getting a pointer to every  Clojure data structure from a nested structure (without taking care of it's aforementioned internals, though).

I think the best way would be to count objects with VisualVM or some other profiler, though.

torsdag 17 juli 2014

Voyuerism in Datomic

Datomic is powerful. Did you know you can see everything you have ever done to your database?

(ns datomic.voyuer
  (:require [datomic.api :as d :refer [db q]]
            [clojure.pprint :refer [pprint]]))

(def uri "datomic:mem://showoff")

(def schema [{:db/id (d/tempid :db.part/db)
               :db/ident :person/name
               :db/valueType :db.type/string
               :db/cardinality :db.cardinality/one
               :db/doc "A persons name"
               :db.install/_attribute :db.part/db}
              {:db/id (d/tempid :db.part/db)
               :db/ident :person/email
               :db/unique :db.unique/identity
               :db/valueType :db.type/string
               :db/cardinality :db.cardinality/one
               :db/doc "A persons email, must be unique"
               :db.install/_attribute :db.part/db}])


(d/create-database uri)
(def conn (d/connect uri))
(d/transact conn schema)

(d/transact conn [{:db/id (d/tempid :db.part/user)
                   :person/name "Linus"
                   :person/email "itisnot@linusericsson.se"}])

(pprint (seq (d/datoms (db conn) :eavt)))


And the result is a sequence of Datoms which can be seen as entities.

These datoms can be decoded as

[Entity id , Attribute (as a pointer to the attributes entity id), Value (as a clojure literal), Tx (as a pointer to the tx-entity), boolean added?].

I did just add empty lines between the datom-numbers and some titles where apropiate. Everything from the system partition the added schema and data is in the list. It's even well documented!

(#datom[0 10 :db.part/db 13194139533312 true] ;; datom 0 is the System partition
 #datom[0 11 0 13194139533366 true] ;;
13194139533366 a txInstant @ beginning of epoch
 #datom[0 11 3 13194139533366 true]
 #datom[0 11 4 13194139533366 true]
 #datom[0 12 20 13194139533366 true]
 #datom[0 12 21 13194139533366 true]
 #datom[0 12 22 13194139533366 true]
 #datom[0 12 23 13194139533366 true]
 #datom[0 12 24 13194139533366 true]
 #datom[0 12 25 13194139533366 true]
 #datom[0 12 26 13194139533366 true]
 #datom[0 12 27 13194139533366 true]
 #datom[0 12 56 13194139533368 true]
 #datom[0 12 57 13194139533368 true]
 #datom[0 12 58 13194139533368 true]
 #datom[0 12 59 13194139533368 true]
 #datom[0 12 60 13194139533368 true]
 #datom[0 12 61 13194139533368 true]
 #datom[0 13 10 13194139533366 true]
 #datom[0 13 11 13194139533366 true]
 #datom[0 13 12 13194139533366 true]
 #datom[0 13 13 13194139533366 true]
 #datom[0 13 14 13194139533366 true]
 #datom[0 13 15 13194139533366 true]
 #datom[0 13 16 13194139533366 true]
 #datom[0 13 17 13194139533366 true]
 #datom[0 13 18 13194139533366 true]
 #datom[0 13 19 13194139533366 true]
 #datom[0 13 39 13194139533366 true]
 #datom[0 13 40 13194139533366 true]
 #datom[0 13 41 13194139533366 true]
 #datom[0 13 42 13194139533366 true]
 #datom[0 13 43 13194139533366 true]
 #datom[0 13 44 13194139533366 true]
 #datom[0 13 45 13194139533366 true]
 #datom[0 13 46 13194139533366 true]
 #datom[0 13 47 13194139533366 true]
 #datom[0 13 50 13194139533366 true]
 #datom[0 13 51 13194139533366 true]
 #datom[0 13 52 13194139533366 true]
 #datom[0 13 62 13194139533368 true]
 #datom[0 13 63 13194139534312 true]
 #datom[0 13 64 13194139534312 true]
 #datom[0 14 54 13194139533366 true]
 #datom[0 14 55 13194139533366 true]
 #datom[0 62 "Name of the system partition. The system partition includes the core of datomic, as well as user schemas: type definitions, attribute definitions, partition definitions, and data function definitions." 13194139533375 true]



 #datom[1 10 :db/add 13194139533312 true]
 #datom[1 62 "Primitive assertion. All transactions eventually reduce to a collection of primitive assertions
 and retractions of facts, e.g. [:db/add fred :age 42]." 13194139533375 true]


 #datom[2 10 :db/retract 13194139533312 true]
 #datom[2 62 "Primitive retraction. All transactions eventually reduce to a collection of assertions and retractions of facts, e.g. [:db/retract fred :age 42]." 13194139533375 true]


 #datom[3 10 :db.part/tx 13194139533312 true]
 #datom[3 62 "Partition used to store data about transactions. Transaction data always includes a :db/txInstant which is the transaction's timestamp, and can be extended to store other information at transaction granularity." 13194139533375 true]


 #datom[4 10 :db.part/user 13194139533312 true]
 #datom[4 62 "Name of the user partition. The user partition is analogous to the default namespace in a programming language, and should be used as a temporary home for data during interactive development." 13194139533375 true]


 #datom[10 10 :db/ident 13194139533312 true]
 #datom[10 40 21 13194139533366 true]
 #datom[10 41 35 13194139533366 true]
 #datom[10 42 38 13194139533366 true]
 #datom[10 62 "Attribute used to uniquely name an entity." 13194139533375 true]


 #datom[11 10 :db.install/partition 13194139533312 true]
 #datom[11 40 20 13194139533366 true]
 #datom[11 41 36 13194139533366 true]
 #datom[11 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a partition." 13194139533375 true]


 #datom[12 10 :db.install/valueType 13194139533312 true]
 #datom[12 40 20 13194139533366 true]
 #datom[12 41 36 13194139533366 true]
 #datom[12 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a value type." 13194139533375 true]


 #datom[13 10 :db.install/attribute 13194139533312 true]
 #datom[13 40 20 13194139533366 true]
 #datom[13 41 36 13194139533366 true]
 #datom[13 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as an attribute." 13194139533375 true]


 #datom[14 10 :db.install/function 13194139533312 true]
 #datom[14 40 20 13194139533366 true]
 #datom[14 41 36 13194139533366 true]
 #datom[14 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a data function." 13194139533375 true]


 #datom[15 10 :db/excise 13194139533312 true]
 #datom[15 40 20 13194139533366 true]
 #datom[15 41 35 13194139533366 true]


 #datom[16 10 :db.excise/attrs 13194139533312 true]
 #datom[16 40 20 13194139533366 true]
 #datom[16 41 36 13194139533366 true]


 #datom[17 10 :db.excise/beforeT 13194139533312 true]
 #datom[17 40 22 13194139533366 true]
 #datom[17 41 35 13194139533366 true]


 #datom[18 10 :db.excise/before 13194139533312 true]
 #datom[18 40 25 13194139533366 true]
 #datom[18 41 35 13194139533366 true]


 #datom[19 10 :db.alter/attribute 13194139533312 true]
 #datom[19 40 20 13194139533366 true]
 #datom[19 41 36 13194139533366 true]
 #datom[19 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will alter the definition o
f existing attribute v." 13194139533375 true]


 #datom[20 10 :db.type/ref 13194139533312 true]
 #datom[20 39 :ref 13194139533366 true]
 #datom[20 62 "Value type for references. All references from one entity to another are through attributes with this value type." 13194139533375 true]


 #datom[21 10 :db.type/keyword 13194139533312 true]
 #datom[21 39 :key 13194139533366 true]
 #datom[21 62 "Value type for keywords. Keywords are used as names, and are interned for efficiency. Keywords map to the native interned-name type in languages that support them." 13194139533375 true]


 #datom[22 10 :db.type/long 13194139533312 true]
 #datom[22 39 :int 13194139533366 true]
 #datom[22 62 "Fixed integer value type. Same semantics as a Java long: 64 bits wide, two's complement binary representation." 13194139533375 true]


 #datom[23 10 :db.type/string 13194139533312 true]
 #datom[23 39 :string 13194139533366 true]
 #datom[23 62 "Value type for strings." 13194139533375 true]


 #datom[24 10 :db.type/boolean 13194139533312 true]
 #datom[24 39 :bool 13194139533366 true]
 #datom[24 62 "Boolean value type." 13194139533375 true]


 #datom[25 10 :db.type/instant 13194139533312 true]
 #datom[25 39 :inst 13194139533366 true]
 #datom[25 62 "Value type for instants in time. Stored internally as a number of milliseconds since midnight, January 1, 1970 UTC. Representation type will vary depending on the language you are using." 13194139533375 true]


 #datom[26 10 :db.type/fn 13194139533312 true]
 #datom[26 39 :datomic/fn 13194139533366 true]
 #datom[26 62 "Value type for database functions. See Javadoc for Peer.function." 13194139533375 true]


 #datom[27 10 :db.type/bytes 13194139533312 true]
 #datom[27 39 :bytes 13194139533366 true]
 #datom[27 62 "Value type for small binaries. Maps to byte array on the JVM." 13194139533375 true]


 #datom[35 10 :db.cardinality/one 13194139533312 true]
 #datom[35 62 "One of two legal values for the :db/cardinality attribute. Specify :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued a
ttributes." 13194139533375 true]


 #datom[36 10 :db.cardinality/many 13194139533312 true]
 #datom[36 62 "One of two legal values for the :db/cardinality attribute. Specify :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes." 13194139533375 true]


 #datom[37 10 :db.unique/value 13194139533312 true]
 #datom[37 62 "Specifies that an attribute's value is unique. Attempts to create a new entity with a colliding value for a :db.unique/value will fail." 13194139533375 true]


 #datom[38 10 :db.unique/identity 13194139533312 true]
 #datom[38 62 "Specifies that an attribute's value is unique. Attempts to create a new entity with a colliding value for a :db.unique/value will become upserts." 13194139533375 true]


 #datom[39 10 :fressian/tag 13194139533312 true]
 #datom[39 40 21 13194139533366 true]
 #datom[39 41 35 13194139533366 true]
 #datom[39 44 true 13194139533366 true]
 #datom[39 62 "Keyword-valued attribute of a value type that specifies the underlying fressian type
used for serialization." 13194139533375 true]


 #datom[40 10 :db/valueType 13194139533312 true]
 #datom[40 40 20 13194139533366 true]
 #datom[40 41 35 13194139533366 true]
 #datom[40 62 "Property of an attribute that specifies the attribute's value type. Built-in value types include, :db.type/keyword, :db.type/string, :db.type/ref, :db.type/instant, :db.type/long, :db.type/bigdec, :db.type/boolean, :db.type/float, :db.type/uuid, :db.type/double, :db.type/bigint,  :db.type/uri." 13194139533375 true]


 #datom[41 10 :db/cardinality 13194139533312 true]
 #datom[41 40 20 13194139533366 true]
 #datom[41 41 35 13194139533366 true]
 #datom[41 62 "Property of an attribute. Two possible values: :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes. Defaults to :db.cardinality/one." 13194139533375 true]


 #datom[42 10 :db/unique 13194139533312 true]
 #datom[42 40 20 13194139533366 true]
 #datom[42 41 35 13194139533366 true]
 #datom[42 62 "Property of an attribute. If value is
 :db.unique/value, then attribute value is unique to each entity. Attempts to insert a duplicate value for a temporary entity id will fail. If value is :db.unique/identity, then attribute value is unique, and upsert is enabled. Attempting to insert a duplicate value for a temporary entity id will cause all attributes associated with that temporary id to be merged with the entity already in the database. Defaults to nil." 13194139533375 true]


 #datom[43 10 :db/isComponent 13194139533312 true]
 #datom[43 40 24 13194139533366 true]
 #datom[43 41 35 13194139533366 true]
 #datom[43 62 "Property of attribute whose vtype is :db.type/ref. If true, then the attribute is a component of the entity referencing it. When you query for an entire entity, components are fetched automatically. Defaults to nil." 13194139533375 true]


 #datom[44 10 :db/index 13194139533312 true]
 #datom[44 40 24 13194139533366 true]
 #datom[44 41 35 13194139533366 true]
 #datom[44 62 "Property of an attribute. If true, create an AVET index for th
e attribute. Defaults to false." 13194139533375 true]


 #datom[45 10 :db/noHistory 13194139533312 true]
 #datom[45 40 24 13194139533366 true]
 #datom[45 41 35 13194139533366 true]
 #datom[45 62 "Property of an attribute. If true, past values of the attribute are not retained after indexing. Defaults to false." 13194139533375 true]


 #datom[46 10 :db/lang 13194139533312 true]
 #datom[46 40 20 13194139533366 true]
 #datom[46 41 35 13194139533366 true]
 #datom[46 62 "Attribute of a data function. Value is a keyword naming the implementation language of the function. Legal values are :db.lang/java and :db.lang/clojure" 13194139533375 true]


 #datom[47 10 :db/code 13194139533312 true]
 #datom[47 40 23 13194139533366 true]
 #datom[47 41 35 13194139533366 true]
 #datom[47 51 true 13194139533366 true]
 #datom[47 62 "String-valued attribute of a data function that contains the function's source code." 13194139533375 true]


 #datom[48 10 :db.lang/clojure 13194139533312 true]
 #datom[48 62 "Value of :db/lang attribute, spec
ifying that a data function is implemented in Clojure." 13194139533375 true]


 #datom[49 10 :db.lang/java 13194139533312 true]
 #datom[49 62 "Value of :db/lang attribute, specifying that a data function is implemented in Java." 13194139533375 true]


 #datom[50 10 :db/txInstant 13194139533312 true]
 #datom[50 40 25 13194139533366 true]
 #datom[50 41 35 13194139533366 true]
 #datom[50 44 true 13194139533366 true]
 #datom[50 62 "Attribute whose value is a :db.type/instant. A :db/txInstant is recorded automatically with every transaction." 13194139533375 true]


 #datom[51 10 :db/fulltext 13194139533312 true]
 #datom[51 40 24 13194139533366 true]
 #datom[51 41 35 13194139533366 true]
 #datom[51 62 "Property of an attribute. If true, create a fulltext search index for the attribute. Defaults to false." 13194139533375 true]


 #datom[52 10 :db/fn 13194139533312 true]
 #datom[52 40 26 13194139533366 true]
 #datom[52 41 35 13194139533366 true]
 #datom[52 62 "A function-valued attribute for direct use by transactions and qu
eries." 13194139533375 true]


 #datom[53 10 :db.bootstrap/part 13194139533312 true]


 #datom[54 10 :db.fn/retractEntity 13194139533366 true]
 #datom[54 46 48 13194139533366 true]
 #datom[54 47 "(clojure.core/fn [db e] (datomic.builtins/build-retract-args db e))" 13194139533366 true]
 #datom[54 62 "Retract all facts about an entity, including references from other entities and component attributes recursively." 13194139533375 true]


 #datom[55 10 :db.fn/cas 13194139533366 true]
 #datom[55 46 48 13194139533366 true]
 #datom[55 47 "(clojure.core/fn [db e a ov nv] (datomic.builtins/compare-and-swap db e a ov nv))" 13194139533366 true]
 #datom[55 62 "Compare and swap the value of an entity's attribute." 13194139533375 true]

Database value types: 

#datom[56 10 :db.type/uuid 13194139533368 true]
 #datom[56 39 :uuid 13194139533368 true]
 #datom[56 62 "Value type for UUIDs. Maps to java.util.UUID on the JVM." 13194139533375 true]


 #datom[57 10 :db.type/double 13194139533368 true]
 #datom[57 39 :double 13194139533368 true]
 #datom[57 62 "Float
ing point value type. Same semantics as a Java double: double-precision 64-bit IEEE 754 floating point." 13194139533375 true]


 #datom[58 10 :db.type/float 13194139533368 true]
 #datom[58 39 :float 13194139533368 true]
 #datom[58 62 "Floating point value type. Same semantics as a Java float: single-precision 32-bit IEEE 754 floating point." 13194139533375 true]


 #datom[59 10 :db.type/uri 13194139533368 true]
 #datom[59 39 :uri 13194139533368 true]
 #datom[59 62 "Value type for URIs. Maps to java.net.URI on the JVM." 13194139533375 true]


 #datom[60 10 :db.type/bigint 13194139533368 true]
 #datom[60 39 :bigint 13194139533368 true]
 #datom[60 62 "Value type for arbitrary precision integers. Maps to java.math.BigInteger on the JVM." 13194139533375 true]


 #datom[61 10 :db.type/bigdec 13194139533368 true]
 #datom[61 39 :bigdec 13194139533368 true]
 #datom[61 62 "Value type for arbitrary precision floating point numbers. Maps to java.math.BigDecimal on the JVM." 13194139533375 true]


 #datom[62 10 :db/doc 13194139533368 true]
 #datom[62 40 23 13194139533368 true]
 #datom[62 41 35 13194139533368 true]
 #datom[62 51 true 13194139533368 true]
 #datom[62 62 "Documentation string for an entity." 13194139533375 true]

The added schema:

 #datom[63 10 :person/name 13194139534312 true]
 #datom[
63 40 23 13194139534312 true]
 #datom[
63 41 35 13194139534312 true]
 #datom[
63 62 "A persons name" 13194139534312 true]

 #datom[64 10 :person/email 13194139534312 true]
 #datom[
64 40 23 13194139534312 true]
 #datom[
64 41 35 13194139534312 true]
 #datom[
64 42 38 13194139534312 true]
 #datom[
64 62 "A persons email, must be unique" 13194139534312 true]

TxInstants, starts at entity id 13194139533366:

 #datom[13194139533366 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533366 true]
 #datom[13194139533368 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533368 true]
 #datom[13194139533375 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533375 true]
 #datom[13194139534312 50 #inst "2014-07-17T16:30:23.535-00:00" 13194139534312 true]
 #datom[13194139534313 50 #inst "2014-07-17T16:30:23.541-00:00" 13194139534313 true]

Added facts

 #datom[17592186045418 63 "Linus" 13194139534313 true]
 #datom[17592186045418 64 "itisnot@linusericsson.se" 13194139534313 true])