måndag 29 december 2014
Limit in Datomic
It's possible to use limit (SQL-style) in Datomic when using the new Pull API. The Pull API seems to be a very ambitious API which can even make graph-walks and more.
torsdag 23 oktober 2014
Talk about a web application architechture
This Monday I held a talk at my company Agical presenting a sketch of an interesting web application architecture. One could call me a Cognitect fan-boy, but I'm proud of it.
Before getting to the technical stuff I discussed a bit on modelling things.
I've always think modelling things are super complicated to get right, especially with ie SQL or single tables like Excel. Probably I try to take to many things into account at the same time, but hey, the world is complicated, so why shouldn't I try to model it at least a bit more like how it really is?
The architechture, then.
Something along these lines: React/Om talking with DataScript via core.async channels and to the server. The server running Pedestal and Datomic (and whatever other data source necessary - Datomic would be the transaction engine).
It's a bit bold to claim that even database queries could be reused in the client, this is likely not the case, but the model could be similar in structure on both client and server.
The controller in the client and server would consist of a core.async pub/sub setup, although it feels a bit hard to make sure everything is setup right, it feels a bit like we are in multimethods all over again.
The start and stop of the application would be taken care of through Component. And off we go!
Extremely useful resources
for preparing this talk was
CatChat (a DataScript example app)
David Nolen's blog posts of both core.async and Om
Om intermediate tutorial on Datomic integration
O'Reilly's Webinar about Pedestal with Ryan Neufeld (2 h YouTube video)
Things I stumbled upon was
Trying to visualize even a simple Pedestal router setup was quite error prone. The interceptors got wrong and all over the place.
Setting up ClojureScript to compile Om. It's mostly the Leiningen project.clj options that's hard to understand. Validation of options or warnings when unused options occur would be useful.
Programming core.async channels in the nRepl often locked up the repl in strange ways, from which I had to re-require the namespaces.
Getting my head around Om and it's quite messy formatting syntax, especially when it comes to catching events from various things. Shouldn't one put small functions inside the elements maps? Maybe listeners should be defined separately from the DOM-structure.
Maybe there's even room for an abstraction over the events, because now most of it is quite OO that one has to get take back tree-element by element.
There are many fascinating papers about CRDT and similar techniques, but still it's not clear on how to structure the events from the application and the event "schema" between client and server, especially with regards to CRDT at similar. An especially delicate thing close to my heart is the ability to show where other people are editing, so edits not collide to badly.
Pub/sub "reactions" I think could be replaced with transducers.
I've always think modelling things are super complicated to get right, especially with ie SQL or single tables like Excel. Probably I try to take to many things into account at the same time, but hey, the world is complicated, so why shouldn't I try to model it at least a bit more like how it really is?
The architechture, then.
Something along these lines: React/Om talking with DataScript via core.async channels and to the server. The server running Pedestal and Datomic (and whatever other data source necessary - Datomic would be the transaction engine).
It's a bit bold to claim that even database queries could be reused in the client, this is likely not the case, but the model could be similar in structure on both client and server.
The controller in the client and server would consist of a core.async pub/sub setup, although it feels a bit hard to make sure everything is setup right, it feels a bit like we are in multimethods all over again.
The start and stop of the application would be taken care of through Component. And off we go!
Extremely useful resources
for preparing this talk was
CatChat (a DataScript example app)
David Nolen's blog posts of both core.async and Om
Om intermediate tutorial on Datomic integration
O'Reilly's Webinar about Pedestal with Ryan Neufeld (2 h YouTube video)
Things I stumbled upon was
Trying to visualize even a simple Pedestal router setup was quite error prone. The interceptors got wrong and all over the place.
Setting up ClojureScript to compile Om. It's mostly the Leiningen project.clj options that's hard to understand. Validation of options or warnings when unused options occur would be useful.
Programming core.async channels in the nRepl often locked up the repl in strange ways, from which I had to re-require the namespaces.
Getting my head around Om and it's quite messy formatting syntax, especially when it comes to catching events from various things. Shouldn't one put small functions inside the elements maps? Maybe listeners should be defined separately from the DOM-structure.
Maybe there's even room for an abstraction over the events, because now most of it is quite OO that one has to get take back tree-element by element.
There are many fascinating papers about CRDT and similar techniques, but still it's not clear on how to structure the events from the application and the event "schema" between client and server, especially with regards to CRDT at similar. An especially delicate thing close to my heart is the ability to show where other people are editing, so edits not collide to badly.
Pub/sub "reactions" I think could be replaced with transducers.
måndag 6 oktober 2014
Clojure web security
This post about Clojure web security is a must read. Not only does it summarize many things that can go bad, but also shows just how severe it can be to read data with read-string - it looks like it can execute almost any code and construct any availiable java class! Scarier than I knew. Thanks.
tisdag 12 augusti 2014
Steam-roller for nested Clojure data structures
There was a question on the Clojure Google group about structural sharing in rrb-vectors visavi a zipper-construction.
As we all know, it's not very easy to be certain of how much memory is allocated in Java/Clojure since the pointers are sometimes reused, and the situation gets even more complicated (for the better) when we have structural sharing and immutable objects.
The Clojure data structures are constructed from Object arrays, which creates shallow nested trees of either vector elements or key-value pairs in the case of PersistentMaps. Likely one could use reflection to dig deeper into the the inner parts of these data structures and arrays, Iroh and Clj-wallhack seems both worthy to try out.
I created a small function for getting a pointer to every Clojure data structure from a nested structure (without taking care of it's aforementioned internals, though).
I think the best way would be to count objects with VisualVM or some other profiler, though.
As we all know, it's not very easy to be certain of how much memory is allocated in Java/Clojure since the pointers are sometimes reused, and the situation gets even more complicated (for the better) when we have structural sharing and immutable objects.
The Clojure data structures are constructed from Object arrays, which creates shallow nested trees of either vector elements or key-value pairs in the case of PersistentMaps. Likely one could use reflection to dig deeper into the the inner parts of these data structures and arrays, Iroh and Clj-wallhack seems both worthy to try out.
I created a small function for getting a pointer to every Clojure data structure from a nested structure (without taking care of it's aforementioned internals, though).
I think the best way would be to count objects with VisualVM or some other profiler, though.
torsdag 17 juli 2014
Voyuerism in Datomic
Datomic is powerful. Did you know you can see everything you have ever done to your database?
(ns datomic.voyuer
(:require [datomic.api :as d :refer [db q]]
[clojure.pprint :refer [pprint]]))
(def uri "datomic:mem://showoff")
(def schema [{:db/id (d/tempid :db.part/db)
:db/ident :person/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A persons name"
:db.install/_attribute :db.part/db}
{:db/id (d/tempid :db.part/db)
:db/ident :person/email
:db/unique :db.unique/identity
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A persons email, must be unique"
:db.install/_attribute :db.part/db}])
(d/create-database uri)
(def conn (d/connect uri))
(d/transact conn schema)
(d/transact conn [{:db/id (d/tempid :db.part/user)
:person/name "Linus"
:person/email "itisnot@linusericsson.se"}])
(pprint (seq (d/datoms (db conn) :eavt)))
And the result is a sequence of Datoms which can be seen as entities.
These datoms can be decoded as
[Entity id , Attribute (as a pointer to the attributes entity id), Value (as a clojure literal), Tx (as a pointer to the tx-entity), boolean added?].
I did just add empty lines between the datom-numbers and some titles where apropiate. Everything from the system partition the added schema and data is in the list. It's even well documented!
(#datom[0 10 :db.part/db 13194139533312 true] ;; datom 0 is the System partition
#datom[0 11 0 13194139533366 true] ;;13194139533366 a txInstant @ beginning of epoch
#datom[0 11 3 13194139533366 true]
#datom[0 11 4 13194139533366 true]
#datom[0 12 20 13194139533366 true]
#datom[0 12 21 13194139533366 true]
#datom[0 12 22 13194139533366 true]
#datom[0 12 23 13194139533366 true]
#datom[0 12 24 13194139533366 true]
#datom[0 12 25 13194139533366 true]
#datom[0 12 26 13194139533366 true]
#datom[0 12 27 13194139533366 true]
#datom[0 12 56 13194139533368 true]
#datom[0 12 57 13194139533368 true]
#datom[0 12 58 13194139533368 true]
#datom[0 12 59 13194139533368 true]
#datom[0 12 60 13194139533368 true]
#datom[0 12 61 13194139533368 true]
#datom[0 13 10 13194139533366 true]
#datom[0 13 11 13194139533366 true]
#datom[0 13 12 13194139533366 true]
#datom[0 13 13 13194139533366 true]
#datom[0 13 14 13194139533366 true]
#datom[0 13 15 13194139533366 true]
#datom[0 13 16 13194139533366 true]
#datom[0 13 17 13194139533366 true]
#datom[0 13 18 13194139533366 true]
#datom[0 13 19 13194139533366 true]
#datom[0 13 39 13194139533366 true]
#datom[0 13 40 13194139533366 true]
#datom[0 13 41 13194139533366 true]
#datom[0 13 42 13194139533366 true]
#datom[0 13 43 13194139533366 true]
#datom[0 13 44 13194139533366 true]
#datom[0 13 45 13194139533366 true]
#datom[0 13 46 13194139533366 true]
#datom[0 13 47 13194139533366 true]
#datom[0 13 50 13194139533366 true]
#datom[0 13 51 13194139533366 true]
#datom[0 13 52 13194139533366 true]
#datom[0 13 62 13194139533368 true]
#datom[0 13 63 13194139534312 true]
#datom[0 13 64 13194139534312 true]
#datom[0 14 54 13194139533366 true]
#datom[0 14 55 13194139533366 true]
#datom[0 62 "Name of the system partition. The system partition includes the core of datomic, as well as user schemas: type definitions, attribute definitions, partition definitions, and data function definitions." 13194139533375 true]
#datom[1 10 :db/add 13194139533312 true]
#datom[1 62 "Primitive assertion. All transactions eventually reduce to a collection of primitive assertions
and retractions of facts, e.g. [:db/add fred :age 42]." 13194139533375 true]
#datom[2 10 :db/retract 13194139533312 true]
#datom[2 62 "Primitive retraction. All transactions eventually reduce to a collection of assertions and retractions of facts, e.g. [:db/retract fred :age 42]." 13194139533375 true]
#datom[3 10 :db.part/tx 13194139533312 true]
#datom[3 62 "Partition used to store data about transactions. Transaction data always includes a :db/txInstant which is the transaction's timestamp, and can be extended to store other information at transaction granularity." 13194139533375 true]
#datom[4 10 :db.part/user 13194139533312 true]
#datom[4 62 "Name of the user partition. The user partition is analogous to the default namespace in a programming language, and should be used as a temporary home for data during interactive development." 13194139533375 true]
#datom[10 10 :db/ident 13194139533312 true]
#datom[10 40 21 13194139533366 true]
#datom[10 41 35 13194139533366 true]
#datom[10 42 38 13194139533366 true]
#datom[10 62 "Attribute used to uniquely name an entity." 13194139533375 true]
#datom[11 10 :db.install/partition 13194139533312 true]
#datom[11 40 20 13194139533366 true]
#datom[11 41 36 13194139533366 true]
#datom[11 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a partition." 13194139533375 true]
#datom[12 10 :db.install/valueType 13194139533312 true]
#datom[12 40 20 13194139533366 true]
#datom[12 41 36 13194139533366 true]
#datom[12 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a value type." 13194139533375 true]
#datom[13 10 :db.install/attribute 13194139533312 true]
#datom[13 40 20 13194139533366 true]
#datom[13 41 36 13194139533366 true]
#datom[13 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as an attribute." 13194139533375 true]
#datom[14 10 :db.install/function 13194139533312 true]
#datom[14 40 20 13194139533366 true]
#datom[14 41 36 13194139533366 true]
#datom[14 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a data function." 13194139533375 true]
#datom[15 10 :db/excise 13194139533312 true]
#datom[15 40 20 13194139533366 true]
#datom[15 41 35 13194139533366 true]
#datom[16 10 :db.excise/attrs 13194139533312 true]
#datom[16 40 20 13194139533366 true]
#datom[16 41 36 13194139533366 true]
#datom[17 10 :db.excise/beforeT 13194139533312 true]
#datom[17 40 22 13194139533366 true]
#datom[17 41 35 13194139533366 true]
#datom[18 10 :db.excise/before 13194139533312 true]
#datom[18 40 25 13194139533366 true]
#datom[18 41 35 13194139533366 true]
#datom[19 10 :db.alter/attribute 13194139533312 true]
#datom[19 40 20 13194139533366 true]
#datom[19 41 36 13194139533366 true]
#datom[19 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will alter the definition o
f existing attribute v." 13194139533375 true]
#datom[20 10 :db.type/ref 13194139533312 true]
#datom[20 39 :ref 13194139533366 true]
#datom[20 62 "Value type for references. All references from one entity to another are through attributes with this value type." 13194139533375 true]
#datom[21 10 :db.type/keyword 13194139533312 true]
#datom[21 39 :key 13194139533366 true]
#datom[21 62 "Value type for keywords. Keywords are used as names, and are interned for efficiency. Keywords map to the native interned-name type in languages that support them." 13194139533375 true]
#datom[22 10 :db.type/long 13194139533312 true]
#datom[22 39 :int 13194139533366 true]
#datom[22 62 "Fixed integer value type. Same semantics as a Java long: 64 bits wide, two's complement binary representation." 13194139533375 true]
#datom[23 10 :db.type/string 13194139533312 true]
#datom[23 39 :string 13194139533366 true]
#datom[23 62 "Value type for strings." 13194139533375 true]
#datom[24 10 :db.type/boolean 13194139533312 true]
#datom[24 39 :bool 13194139533366 true]
#datom[24 62 "Boolean value type." 13194139533375 true]
#datom[25 10 :db.type/instant 13194139533312 true]
#datom[25 39 :inst 13194139533366 true]
#datom[25 62 "Value type for instants in time. Stored internally as a number of milliseconds since midnight, January 1, 1970 UTC. Representation type will vary depending on the language you are using." 13194139533375 true]
#datom[26 10 :db.type/fn 13194139533312 true]
#datom[26 39 :datomic/fn 13194139533366 true]
#datom[26 62 "Value type for database functions. See Javadoc for Peer.function." 13194139533375 true]
#datom[27 10 :db.type/bytes 13194139533312 true]
#datom[27 39 :bytes 13194139533366 true]
#datom[27 62 "Value type for small binaries. Maps to byte array on the JVM." 13194139533375 true]
#datom[35 10 :db.cardinality/one 13194139533312 true]
#datom[35 62 "One of two legal values for the :db/cardinality attribute. Specify :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued a
ttributes." 13194139533375 true]
#datom[36 10 :db.cardinality/many 13194139533312 true]
#datom[36 62 "One of two legal values for the :db/cardinality attribute. Specify :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes." 13194139533375 true]
#datom[37 10 :db.unique/value 13194139533312 true]
#datom[37 62 "Specifies that an attribute's value is unique. Attempts to create a new entity with a colliding value for a :db.unique/value will fail." 13194139533375 true]
#datom[38 10 :db.unique/identity 13194139533312 true]
#datom[38 62 "Specifies that an attribute's value is unique. Attempts to create a new entity with a colliding value for a :db.unique/value will become upserts." 13194139533375 true]
#datom[39 10 :fressian/tag 13194139533312 true]
#datom[39 40 21 13194139533366 true]
#datom[39 41 35 13194139533366 true]
#datom[39 44 true 13194139533366 true]
#datom[39 62 "Keyword-valued attribute of a value type that specifies the underlying fressian type
used for serialization." 13194139533375 true]
#datom[40 10 :db/valueType 13194139533312 true]
#datom[40 40 20 13194139533366 true]
#datom[40 41 35 13194139533366 true]
#datom[40 62 "Property of an attribute that specifies the attribute's value type. Built-in value types include, :db.type/keyword, :db.type/string, :db.type/ref, :db.type/instant, :db.type/long, :db.type/bigdec, :db.type/boolean, :db.type/float, :db.type/uuid, :db.type/double, :db.type/bigint, :db.type/uri." 13194139533375 true]
#datom[41 10 :db/cardinality 13194139533312 true]
#datom[41 40 20 13194139533366 true]
#datom[41 41 35 13194139533366 true]
#datom[41 62 "Property of an attribute. Two possible values: :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes. Defaults to :db.cardinality/one." 13194139533375 true]
#datom[42 10 :db/unique 13194139533312 true]
#datom[42 40 20 13194139533366 true]
#datom[42 41 35 13194139533366 true]
#datom[42 62 "Property of an attribute. If value is
:db.unique/value, then attribute value is unique to each entity. Attempts to insert a duplicate value for a temporary entity id will fail. If value is :db.unique/identity, then attribute value is unique, and upsert is enabled. Attempting to insert a duplicate value for a temporary entity id will cause all attributes associated with that temporary id to be merged with the entity already in the database. Defaults to nil." 13194139533375 true]
#datom[43 10 :db/isComponent 13194139533312 true]
#datom[43 40 24 13194139533366 true]
#datom[43 41 35 13194139533366 true]
#datom[43 62 "Property of attribute whose vtype is :db.type/ref. If true, then the attribute is a component of the entity referencing it. When you query for an entire entity, components are fetched automatically. Defaults to nil." 13194139533375 true]
#datom[44 10 :db/index 13194139533312 true]
#datom[44 40 24 13194139533366 true]
#datom[44 41 35 13194139533366 true]
#datom[44 62 "Property of an attribute. If true, create an AVET index for th
e attribute. Defaults to false." 13194139533375 true]
#datom[45 10 :db/noHistory 13194139533312 true]
#datom[45 40 24 13194139533366 true]
#datom[45 41 35 13194139533366 true]
#datom[45 62 "Property of an attribute. If true, past values of the attribute are not retained after indexing. Defaults to false." 13194139533375 true]
#datom[46 10 :db/lang 13194139533312 true]
#datom[46 40 20 13194139533366 true]
#datom[46 41 35 13194139533366 true]
#datom[46 62 "Attribute of a data function. Value is a keyword naming the implementation language of the function. Legal values are :db.lang/java and :db.lang/clojure" 13194139533375 true]
#datom[47 10 :db/code 13194139533312 true]
#datom[47 40 23 13194139533366 true]
#datom[47 41 35 13194139533366 true]
#datom[47 51 true 13194139533366 true]
#datom[47 62 "String-valued attribute of a data function that contains the function's source code." 13194139533375 true]
#datom[48 10 :db.lang/clojure 13194139533312 true]
#datom[48 62 "Value of :db/lang attribute, spec
ifying that a data function is implemented in Clojure." 13194139533375 true]
#datom[49 10 :db.lang/java 13194139533312 true]
#datom[49 62 "Value of :db/lang attribute, specifying that a data function is implemented in Java." 13194139533375 true]
#datom[50 10 :db/txInstant 13194139533312 true]
#datom[50 40 25 13194139533366 true]
#datom[50 41 35 13194139533366 true]
#datom[50 44 true 13194139533366 true]
#datom[50 62 "Attribute whose value is a :db.type/instant. A :db/txInstant is recorded automatically with every transaction." 13194139533375 true]
#datom[51 10 :db/fulltext 13194139533312 true]
#datom[51 40 24 13194139533366 true]
#datom[51 41 35 13194139533366 true]
#datom[51 62 "Property of an attribute. If true, create a fulltext search index for the attribute. Defaults to false." 13194139533375 true]
#datom[52 10 :db/fn 13194139533312 true]
#datom[52 40 26 13194139533366 true]
#datom[52 41 35 13194139533366 true]
#datom[52 62 "A function-valued attribute for direct use by transactions and qu
eries." 13194139533375 true]
#datom[53 10 :db.bootstrap/part 13194139533312 true]
#datom[54 10 :db.fn/retractEntity 13194139533366 true]
#datom[54 46 48 13194139533366 true]
#datom[54 47 "(clojure.core/fn [db e] (datomic.builtins/build-retract-args db e))" 13194139533366 true]
#datom[54 62 "Retract all facts about an entity, including references from other entities and component attributes recursively." 13194139533375 true]
#datom[55 10 :db.fn/cas 13194139533366 true]
#datom[55 46 48 13194139533366 true]
#datom[55 47 "(clojure.core/fn [db e a ov nv] (datomic.builtins/compare-and-swap db e a ov nv))" 13194139533366 true]
#datom[55 62 "Compare and swap the value of an entity's attribute." 13194139533375 true]
#datom[56 39 :uuid 13194139533368 true]
#datom[56 62 "Value type for UUIDs. Maps to java.util.UUID on the JVM." 13194139533375 true]
#datom[57 10 :db.type/double 13194139533368 true]
#datom[57 39 :double 13194139533368 true]
#datom[57 62 "Float
ing point value type. Same semantics as a Java double: double-precision 64-bit IEEE 754 floating point." 13194139533375 true]
#datom[58 10 :db.type/float 13194139533368 true]
#datom[58 39 :float 13194139533368 true]
#datom[58 62 "Floating point value type. Same semantics as a Java float: single-precision 32-bit IEEE 754 floating point." 13194139533375 true]
#datom[59 10 :db.type/uri 13194139533368 true]
#datom[59 39 :uri 13194139533368 true]
#datom[59 62 "Value type for URIs. Maps to java.net.URI on the JVM." 13194139533375 true]
#datom[60 10 :db.type/bigint 13194139533368 true]
#datom[60 39 :bigint 13194139533368 true]
#datom[60 62 "Value type for arbitrary precision integers. Maps to java.math.BigInteger on the JVM." 13194139533375 true]
#datom[61 10 :db.type/bigdec 13194139533368 true]
#datom[61 39 :bigdec 13194139533368 true]
#datom[61 62 "Value type for arbitrary precision floating point numbers. Maps to java.math.BigDecimal on the JVM." 13194139533375 true]
#datom[62 10 :db/doc 13194139533368 true]
#datom[62 40 23 13194139533368 true]
#datom[62 41 35 13194139533368 true]
#datom[62 51 true 13194139533368 true]
#datom[62 62 "Documentation string for an entity." 13194139533375 true]
#datom[63 40 23 13194139534312 true]
#datom[63 41 35 13194139534312 true]
#datom[63 62 "A persons name" 13194139534312 true]
#datom[64 10 :person/email 13194139534312 true]
#datom[64 40 23 13194139534312 true]
#datom[64 41 35 13194139534312 true]
#datom[64 42 38 13194139534312 true]
#datom[64 62 "A persons email, must be unique" 13194139534312 true]
#datom[13194139533368 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533368 true]
#datom[13194139533375 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533375 true]
#datom[13194139534312 50 #inst "2014-07-17T16:30:23.535-00:00" 13194139534312 true]
#datom[13194139534313 50 #inst "2014-07-17T16:30:23.541-00:00" 13194139534313 true]
#datom[17592186045418 64 "itisnot@linusericsson.se" 13194139534313 true])
(ns datomic.voyuer
(:require [datomic.api :as d :refer [db q]]
[clojure.pprint :refer [pprint]]))
(def uri "datomic:mem://showoff")
(def schema [{:db/id (d/tempid :db.part/db)
:db/ident :person/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A persons name"
:db.install/_attribute :db.part/db}
{:db/id (d/tempid :db.part/db)
:db/ident :person/email
:db/unique :db.unique/identity
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A persons email, must be unique"
:db.install/_attribute :db.part/db}])
(d/create-database uri)
(def conn (d/connect uri))
(d/transact conn schema)
(d/transact conn [{:db/id (d/tempid :db.part/user)
:person/name "Linus"
:person/email "itisnot@linusericsson.se"}])
(pprint (seq (d/datoms (db conn) :eavt)))
And the result is a sequence of Datoms which can be seen as entities.
These datoms can be decoded as
[Entity id , Attribute (as a pointer to the attributes entity id), Value (as a clojure literal), Tx (as a pointer to the tx-entity), boolean added?].
I did just add empty lines between the datom-numbers and some titles where apropiate. Everything from the system partition the added schema and data is in the list. It's even well documented!
(#datom[0 10 :db.part/db 13194139533312 true] ;; datom 0 is the System partition
#datom[0 11 0 13194139533366 true] ;;13194139533366 a txInstant @ beginning of epoch
#datom[0 11 3 13194139533366 true]
#datom[0 11 4 13194139533366 true]
#datom[0 12 20 13194139533366 true]
#datom[0 12 21 13194139533366 true]
#datom[0 12 22 13194139533366 true]
#datom[0 12 23 13194139533366 true]
#datom[0 12 24 13194139533366 true]
#datom[0 12 25 13194139533366 true]
#datom[0 12 26 13194139533366 true]
#datom[0 12 27 13194139533366 true]
#datom[0 12 56 13194139533368 true]
#datom[0 12 57 13194139533368 true]
#datom[0 12 58 13194139533368 true]
#datom[0 12 59 13194139533368 true]
#datom[0 12 60 13194139533368 true]
#datom[0 12 61 13194139533368 true]
#datom[0 13 10 13194139533366 true]
#datom[0 13 11 13194139533366 true]
#datom[0 13 12 13194139533366 true]
#datom[0 13 13 13194139533366 true]
#datom[0 13 14 13194139533366 true]
#datom[0 13 15 13194139533366 true]
#datom[0 13 16 13194139533366 true]
#datom[0 13 17 13194139533366 true]
#datom[0 13 18 13194139533366 true]
#datom[0 13 19 13194139533366 true]
#datom[0 13 39 13194139533366 true]
#datom[0 13 40 13194139533366 true]
#datom[0 13 41 13194139533366 true]
#datom[0 13 42 13194139533366 true]
#datom[0 13 43 13194139533366 true]
#datom[0 13 44 13194139533366 true]
#datom[0 13 45 13194139533366 true]
#datom[0 13 46 13194139533366 true]
#datom[0 13 47 13194139533366 true]
#datom[0 13 50 13194139533366 true]
#datom[0 13 51 13194139533366 true]
#datom[0 13 52 13194139533366 true]
#datom[0 13 62 13194139533368 true]
#datom[0 13 63 13194139534312 true]
#datom[0 13 64 13194139534312 true]
#datom[0 14 54 13194139533366 true]
#datom[0 14 55 13194139533366 true]
#datom[0 62 "Name of the system partition. The system partition includes the core of datomic, as well as user schemas: type definitions, attribute definitions, partition definitions, and data function definitions." 13194139533375 true]
#datom[1 10 :db/add 13194139533312 true]
#datom[1 62 "Primitive assertion. All transactions eventually reduce to a collection of primitive assertions
and retractions of facts, e.g. [:db/add fred :age 42]." 13194139533375 true]
#datom[2 10 :db/retract 13194139533312 true]
#datom[2 62 "Primitive retraction. All transactions eventually reduce to a collection of assertions and retractions of facts, e.g. [:db/retract fred :age 42]." 13194139533375 true]
#datom[3 10 :db.part/tx 13194139533312 true]
#datom[3 62 "Partition used to store data about transactions. Transaction data always includes a :db/txInstant which is the transaction's timestamp, and can be extended to store other information at transaction granularity." 13194139533375 true]
#datom[4 10 :db.part/user 13194139533312 true]
#datom[4 62 "Name of the user partition. The user partition is analogous to the default namespace in a programming language, and should be used as a temporary home for data during interactive development." 13194139533375 true]
#datom[10 10 :db/ident 13194139533312 true]
#datom[10 40 21 13194139533366 true]
#datom[10 41 35 13194139533366 true]
#datom[10 42 38 13194139533366 true]
#datom[10 62 "Attribute used to uniquely name an entity." 13194139533375 true]
#datom[11 10 :db.install/partition 13194139533312 true]
#datom[11 40 20 13194139533366 true]
#datom[11 41 36 13194139533366 true]
#datom[11 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a partition." 13194139533375 true]
#datom[12 10 :db.install/valueType 13194139533312 true]
#datom[12 40 20 13194139533366 true]
#datom[12 41 36 13194139533366 true]
#datom[12 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a value type." 13194139533375 true]
#datom[13 10 :db.install/attribute 13194139533312 true]
#datom[13 40 20 13194139533366 true]
#datom[13 41 36 13194139533366 true]
#datom[13 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as an attribute." 13194139533375 true]
#datom[14 10 :db.install/function 13194139533312 true]
#datom[14 40 20 13194139533366 true]
#datom[14 41 36 13194139533366 true]
#datom[14 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will install v as a data function." 13194139533375 true]
#datom[15 10 :db/excise 13194139533312 true]
#datom[15 40 20 13194139533366 true]
#datom[15 41 35 13194139533366 true]
#datom[16 10 :db.excise/attrs 13194139533312 true]
#datom[16 40 20 13194139533366 true]
#datom[16 41 36 13194139533366 true]
#datom[17 10 :db.excise/beforeT 13194139533312 true]
#datom[17 40 22 13194139533366 true]
#datom[17 41 35 13194139533366 true]
#datom[18 10 :db.excise/before 13194139533312 true]
#datom[18 40 25 13194139533366 true]
#datom[18 41 35 13194139533366 true]
#datom[19 10 :db.alter/attribute 13194139533312 true]
#datom[19 40 20 13194139533366 true]
#datom[19 41 36 13194139533366 true]
#datom[19 62 "System attribute with type :db.type/ref. Asserting this attribute on :db.part/db with value v will alter the definition o
f existing attribute v." 13194139533375 true]
#datom[20 10 :db.type/ref 13194139533312 true]
#datom[20 39 :ref 13194139533366 true]
#datom[20 62 "Value type for references. All references from one entity to another are through attributes with this value type." 13194139533375 true]
#datom[21 10 :db.type/keyword 13194139533312 true]
#datom[21 39 :key 13194139533366 true]
#datom[21 62 "Value type for keywords. Keywords are used as names, and are interned for efficiency. Keywords map to the native interned-name type in languages that support them." 13194139533375 true]
#datom[22 10 :db.type/long 13194139533312 true]
#datom[22 39 :int 13194139533366 true]
#datom[22 62 "Fixed integer value type. Same semantics as a Java long: 64 bits wide, two's complement binary representation." 13194139533375 true]
#datom[23 10 :db.type/string 13194139533312 true]
#datom[23 39 :string 13194139533366 true]
#datom[23 62 "Value type for strings." 13194139533375 true]
#datom[24 10 :db.type/boolean 13194139533312 true]
#datom[24 39 :bool 13194139533366 true]
#datom[24 62 "Boolean value type." 13194139533375 true]
#datom[25 10 :db.type/instant 13194139533312 true]
#datom[25 39 :inst 13194139533366 true]
#datom[25 62 "Value type for instants in time. Stored internally as a number of milliseconds since midnight, January 1, 1970 UTC. Representation type will vary depending on the language you are using." 13194139533375 true]
#datom[26 10 :db.type/fn 13194139533312 true]
#datom[26 39 :datomic/fn 13194139533366 true]
#datom[26 62 "Value type for database functions. See Javadoc for Peer.function." 13194139533375 true]
#datom[27 10 :db.type/bytes 13194139533312 true]
#datom[27 39 :bytes 13194139533366 true]
#datom[27 62 "Value type for small binaries. Maps to byte array on the JVM." 13194139533375 true]
#datom[35 10 :db.cardinality/one 13194139533312 true]
#datom[35 62 "One of two legal values for the :db/cardinality attribute. Specify :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued a
ttributes." 13194139533375 true]
#datom[36 10 :db.cardinality/many 13194139533312 true]
#datom[36 62 "One of two legal values for the :db/cardinality attribute. Specify :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes." 13194139533375 true]
#datom[37 10 :db.unique/value 13194139533312 true]
#datom[37 62 "Specifies that an attribute's value is unique. Attempts to create a new entity with a colliding value for a :db.unique/value will fail." 13194139533375 true]
#datom[38 10 :db.unique/identity 13194139533312 true]
#datom[38 62 "Specifies that an attribute's value is unique. Attempts to create a new entity with a colliding value for a :db.unique/value will become upserts." 13194139533375 true]
#datom[39 10 :fressian/tag 13194139533312 true]
#datom[39 40 21 13194139533366 true]
#datom[39 41 35 13194139533366 true]
#datom[39 44 true 13194139533366 true]
#datom[39 62 "Keyword-valued attribute of a value type that specifies the underlying fressian type
used for serialization." 13194139533375 true]
#datom[40 10 :db/valueType 13194139533312 true]
#datom[40 40 20 13194139533366 true]
#datom[40 41 35 13194139533366 true]
#datom[40 62 "Property of an attribute that specifies the attribute's value type. Built-in value types include, :db.type/keyword, :db.type/string, :db.type/ref, :db.type/instant, :db.type/long, :db.type/bigdec, :db.type/boolean, :db.type/float, :db.type/uuid, :db.type/double, :db.type/bigint, :db.type/uri." 13194139533375 true]
#datom[41 10 :db/cardinality 13194139533312 true]
#datom[41 40 20 13194139533366 true]
#datom[41 41 35 13194139533366 true]
#datom[41 62 "Property of an attribute. Two possible values: :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes. Defaults to :db.cardinality/one." 13194139533375 true]
#datom[42 10 :db/unique 13194139533312 true]
#datom[42 40 20 13194139533366 true]
#datom[42 41 35 13194139533366 true]
#datom[42 62 "Property of an attribute. If value is
:db.unique/value, then attribute value is unique to each entity. Attempts to insert a duplicate value for a temporary entity id will fail. If value is :db.unique/identity, then attribute value is unique, and upsert is enabled. Attempting to insert a duplicate value for a temporary entity id will cause all attributes associated with that temporary id to be merged with the entity already in the database. Defaults to nil." 13194139533375 true]
#datom[43 10 :db/isComponent 13194139533312 true]
#datom[43 40 24 13194139533366 true]
#datom[43 41 35 13194139533366 true]
#datom[43 62 "Property of attribute whose vtype is :db.type/ref. If true, then the attribute is a component of the entity referencing it. When you query for an entire entity, components are fetched automatically. Defaults to nil." 13194139533375 true]
#datom[44 10 :db/index 13194139533312 true]
#datom[44 40 24 13194139533366 true]
#datom[44 41 35 13194139533366 true]
#datom[44 62 "Property of an attribute. If true, create an AVET index for th
e attribute. Defaults to false." 13194139533375 true]
#datom[45 10 :db/noHistory 13194139533312 true]
#datom[45 40 24 13194139533366 true]
#datom[45 41 35 13194139533366 true]
#datom[45 62 "Property of an attribute. If true, past values of the attribute are not retained after indexing. Defaults to false." 13194139533375 true]
#datom[46 10 :db/lang 13194139533312 true]
#datom[46 40 20 13194139533366 true]
#datom[46 41 35 13194139533366 true]
#datom[46 62 "Attribute of a data function. Value is a keyword naming the implementation language of the function. Legal values are :db.lang/java and :db.lang/clojure" 13194139533375 true]
#datom[47 10 :db/code 13194139533312 true]
#datom[47 40 23 13194139533366 true]
#datom[47 41 35 13194139533366 true]
#datom[47 51 true 13194139533366 true]
#datom[47 62 "String-valued attribute of a data function that contains the function's source code." 13194139533375 true]
#datom[48 10 :db.lang/clojure 13194139533312 true]
#datom[48 62 "Value of :db/lang attribute, spec
ifying that a data function is implemented in Clojure." 13194139533375 true]
#datom[49 10 :db.lang/java 13194139533312 true]
#datom[49 62 "Value of :db/lang attribute, specifying that a data function is implemented in Java." 13194139533375 true]
#datom[50 10 :db/txInstant 13194139533312 true]
#datom[50 40 25 13194139533366 true]
#datom[50 41 35 13194139533366 true]
#datom[50 44 true 13194139533366 true]
#datom[50 62 "Attribute whose value is a :db.type/instant. A :db/txInstant is recorded automatically with every transaction." 13194139533375 true]
#datom[51 10 :db/fulltext 13194139533312 true]
#datom[51 40 24 13194139533366 true]
#datom[51 41 35 13194139533366 true]
#datom[51 62 "Property of an attribute. If true, create a fulltext search index for the attribute. Defaults to false." 13194139533375 true]
#datom[52 10 :db/fn 13194139533312 true]
#datom[52 40 26 13194139533366 true]
#datom[52 41 35 13194139533366 true]
#datom[52 62 "A function-valued attribute for direct use by transactions and qu
eries." 13194139533375 true]
#datom[53 10 :db.bootstrap/part 13194139533312 true]
#datom[54 10 :db.fn/retractEntity 13194139533366 true]
#datom[54 46 48 13194139533366 true]
#datom[54 47 "(clojure.core/fn [db e] (datomic.builtins/build-retract-args db e))" 13194139533366 true]
#datom[54 62 "Retract all facts about an entity, including references from other entities and component attributes recursively." 13194139533375 true]
#datom[55 10 :db.fn/cas 13194139533366 true]
#datom[55 46 48 13194139533366 true]
#datom[55 47 "(clojure.core/fn [db e a ov nv] (datomic.builtins/compare-and-swap db e a ov nv))" 13194139533366 true]
#datom[55 62 "Compare and swap the value of an entity's attribute." 13194139533375 true]
Database value types:
#datom[56 10 :db.type/uuid 13194139533368 true]
#datom[56 39 :uuid 13194139533368 true]
#datom[56 62 "Value type for UUIDs. Maps to java.util.UUID on the JVM." 13194139533375 true]
#datom[57 10 :db.type/double 13194139533368 true]
#datom[57 39 :double 13194139533368 true]
#datom[57 62 "Float
ing point value type. Same semantics as a Java double: double-precision 64-bit IEEE 754 floating point." 13194139533375 true]
#datom[58 10 :db.type/float 13194139533368 true]
#datom[58 39 :float 13194139533368 true]
#datom[58 62 "Floating point value type. Same semantics as a Java float: single-precision 32-bit IEEE 754 floating point." 13194139533375 true]
#datom[59 10 :db.type/uri 13194139533368 true]
#datom[59 39 :uri 13194139533368 true]
#datom[59 62 "Value type for URIs. Maps to java.net.URI on the JVM." 13194139533375 true]
#datom[60 10 :db.type/bigint 13194139533368 true]
#datom[60 39 :bigint 13194139533368 true]
#datom[60 62 "Value type for arbitrary precision integers. Maps to java.math.BigInteger on the JVM." 13194139533375 true]
#datom[61 10 :db.type/bigdec 13194139533368 true]
#datom[61 39 :bigdec 13194139533368 true]
#datom[61 62 "Value type for arbitrary precision floating point numbers. Maps to java.math.BigDecimal on the JVM." 13194139533375 true]
#datom[62 10 :db/doc 13194139533368 true]
#datom[62 40 23 13194139533368 true]
#datom[62 41 35 13194139533368 true]
#datom[62 51 true 13194139533368 true]
#datom[62 62 "Documentation string for an entity." 13194139533375 true]
The added schema:
#datom[63 10 :person/name 13194139534312 true]#datom[63 40 23 13194139534312 true]
#datom[63 41 35 13194139534312 true]
#datom[63 62 "A persons name" 13194139534312 true]
#datom[64 10 :person/email 13194139534312 true]
#datom[64 40 23 13194139534312 true]
#datom[64 41 35 13194139534312 true]
#datom[64 42 38 13194139534312 true]
#datom[64 62 "A persons email, must be unique" 13194139534312 true]
TxInstants, starts at entity id 13194139533366:
#datom[13194139533366 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533366 true]#datom[13194139533368 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533368 true]
#datom[13194139533375 50 #inst "1970-01-01T00:00:00.000-00:00" 13194139533375 true]
#datom[13194139534312 50 #inst "2014-07-17T16:30:23.535-00:00" 13194139534312 true]
#datom[13194139534313 50 #inst "2014-07-17T16:30:23.541-00:00" 13194139534313 true]
Added facts
#datom[17592186045418 63 "Linus" 13194139534313 true]#datom[17592186045418 64 "itisnot@linusericsson.se" 13194139534313 true])
Spy with Timbre
Peter Taoussanis Timbre logging library is awesome. It's dead easy to get started with - add a dependecy and you're done.
One really usable features in it (apart from the ordinary "write this to log" functions) is spy.
spy let's you wrap an expression and log both the expression and it's result, like so:
(require '[taoensso.timbre :as timbre])
(defn some-function [argh]
(timbre/spy (reverse argh)))
(timbre/spy (reverse argh)))
(defn some-other-function [x]
(timbre/spy (map rand-int x)))
(timbre/spy (map rand-int x)))
(some-other-function (some-function [1 2 34]))
2014-Jul-17 18:12:06 +0200 Albatron-5 DEBUG [user] - (reverse argh) (34 2 1)
2014-Jul-17 18:12:06 +0200 Albatron-5 DEBUG [user] - (map rand-int x) (30 0 0)
-> (30 0 0)
2014-Jul-17 18:12:06 +0200 Albatron-5 DEBUG [user] - (reverse argh) (34 2 1)
2014-Jul-17 18:12:06 +0200 Albatron-5 DEBUG [user] - (map rand-int x) (30 0 0)
-> (30 0 0)
The result of the two above functions are not apparent, and there is good reason to keep an eye on intermediate results. This can be tedious when not in light-table. Not so with spy.
söndag 4 maj 2014
The risks with TDD
Lately there has been a lot of debate about Test Driven Development. For me I started consider the usage of TDD when Rich Hickey formulated it as "you wouldn't drive a car by crashing into the guard-rails". I would change the metaphor slightly to mean "you don't design cars not crash into the guard-rails".
Whether you hit the guard-rails or not is not a decent measure whether a car is good or not. The higher idea of a car is not something that don't crashes into the guard rails. The higher idea of a car is more of something like a tool to transport yourself and some more people and stuff quickly, safe, cheap, and in a joyful manner from where you are now to some place farther away than you could comfortably walk. No guard rails involved.
"But the test-cases is a form of strict requirement specifications!" you say. Of course they are. It's necessary to specify requirements in some form, to detect logic inconsistencies and communicate the design (to make others able to criticize it). Write test cases is a way of doing this, but usually not the best way.
Another problem which is perpendicular to get a logically consistent view of the problem is the problem of avoiding/catch trivial implementation mistakes early. By trivial I mean something like a comparison with > instead of >=. These mistakes usually don't really touch the higher idea of the program, but are of course crucial to get correct. Handmade test cases, asserts, contract programming and even generative testing are great tools for this. Don't mix solving these bugs with detecting higher level logic inconsistencies of your code. Trivial bugs are relieving to correct, but never of any significant value for the program. It's not the hard problem.
My baby message-queue example
The risk with test-driven development is that you start to code before you know what to code. It is of course a good way to get yourself out of analysis paralysis, but in my experience, there are to little 'analysis paralysis' compared to 'cowboy coding that gets impossible to manage and extend later on'.
Let's say you are in need of some kind of messaging system. Your system is spread over several computers, and you have already utilized ad-hoc socket stream formats and various REST-APIs in an unsustainable way, just to get the whole thing going. You shrug at the idea of expanding the system. Wouldn't it be nice to solve this problem once and for all?
Let's say we try to formulate some test cases about how this messaging system could work. Obviously you'll need some way to connect the program to the messaging system bus, and some way to send and receive messages.
It could be a test-case like this:
(let [connection (connect "some-message-bus")]
(is (up? connection)))
and
(let [connection (connect "some-message-bus")]
(send! connection "hello")
(is (== (receive! connection) "hello"))))
This is great! It is simple! I can see that there's a function connect that need to be implemented. This needs to return an object that, given to a function up? returns true.
I also want to be able to send and receive things on each bus.
This is actually highly enlightening. It's a minimalistic interface but it captures a small API. I know what I want and we can even discuss parts of the design given these small lines of code.
Of course there's a ton of various things I need to add - some event loop facility, networking support to be able to connect remote computers, reliance of the message queue, potentially other behaviours, but I could very well code up something that would work quite OK in process just given this simple test. Almost like magic!
Never underestimate the real problem in distributed computing
It could easily be the case that we actually would be in need of quite a different solution than the one we thought we needed - maybe we didn't parallelize the problem enough and did all the socket shuffling in vain (I've seen that more than once). And how do we handle that connections are lost? Can we unsubscribe a bus? What happens if we subscribe to many queues. What if servers are down? New servers join the cluster? Authentication? Am I the first person in the Universe facing this particular problem and is it unique of its kind? (that last one was a rhetoric question).
Depending on the specific problem the computing cluster should solve, there are many different ways to approach it. My small TDD approach has the insidious side-effect that it actually makes me narrow down my view of the problem way too early. It's very hard to kill code you wrote start over, almost from scratch. Good test takes effort to specify (and rightly so). To refactor tests probably takes even more effort than to refactor code.
The TDD way to navigate through the large space of solutions, can easily get stuck on a local maxima.
I can do nothing but approve Rich Hickeys idea of "hammock time". You really have to understand and be able to keep the whole problem you try to solve, and as many of its parameters and quirks. One good way to do this is really to try to solve the problem (preferably on paper) and after that see how others did solve similar problems, code up a small prototype, see how others code would solve the same problem, make sure you know how to the whole stack would work. When this work is done: TDD would work just fine.
TDD is a great scaffold, but a most often a really shitty sketch. Actually test cases have too high fidelity and is to slow to write and change, and you can mistake them for "real test cases that should be used to test the final solution on".
Test cases is not a good blue print either - test cases does not say how the solution should work, only how it should behave. If you really want to code this way (which is really powerful), use a declarative programming language, like datalog instead of abusing your precious brain time with being a manual compiler for your own, made up, non-standard, most likely hard-to-understand declarative programming language. (Those are more common than you might think).
In summary
The major risk with TDD is that one get carried away on the wrong track, starts to solve some problem, get some kick from passing tests, continues in that direction for more passing tests kicks, and never really takes the opportunity to really think and reason systematically about the whole problem one have at hand.
Don't be mean to yourself and your friends, always think hard on your problem and possible solutions AFK before coding. If that is not possible, something is wrong, and your code will likely be as messy as the problem it tries to solve.
When you know exactly what the outlines of the problem are and how you want to solve this problem - then TDD is one of several tools to get your code super duper great. Use it accordingly. Thanks.
Whether you hit the guard-rails or not is not a decent measure whether a car is good or not. The higher idea of a car is not something that don't crashes into the guard rails. The higher idea of a car is more of something like a tool to transport yourself and some more people and stuff quickly, safe, cheap, and in a joyful manner from where you are now to some place farther away than you could comfortably walk. No guard rails involved.
"But the test-cases is a form of strict requirement specifications!" you say. Of course they are. It's necessary to specify requirements in some form, to detect logic inconsistencies and communicate the design (to make others able to criticize it). Write test cases is a way of doing this, but usually not the best way.
Another problem which is perpendicular to get a logically consistent view of the problem is the problem of avoiding/catch trivial implementation mistakes early. By trivial I mean something like a comparison with > instead of >=. These mistakes usually don't really touch the higher idea of the program, but are of course crucial to get correct. Handmade test cases, asserts, contract programming and even generative testing are great tools for this. Don't mix solving these bugs with detecting higher level logic inconsistencies of your code. Trivial bugs are relieving to correct, but never of any significant value for the program. It's not the hard problem.
My baby message-queue example
The risk with test-driven development is that you start to code before you know what to code. It is of course a good way to get yourself out of analysis paralysis, but in my experience, there are to little 'analysis paralysis' compared to 'cowboy coding that gets impossible to manage and extend later on'.
Let's say you are in need of some kind of messaging system. Your system is spread over several computers, and you have already utilized ad-hoc socket stream formats and various REST-APIs in an unsustainable way, just to get the whole thing going. You shrug at the idea of expanding the system. Wouldn't it be nice to solve this problem once and for all?
Let's say we try to formulate some test cases about how this messaging system could work. Obviously you'll need some way to connect the program to the messaging system bus, and some way to send and receive messages.
It could be a test-case like this:
(let [connection (connect "some-message-bus")]
(is (up? connection)))
and
(let [connection (connect "some-message-bus")]
(send! connection "hello")
(is (== (receive! connection) "hello"))))
This is great! It is simple! I can see that there's a function connect that need to be implemented. This needs to return an object that, given to a function up? returns true.
I also want to be able to send and receive things on each bus.
This is actually highly enlightening. It's a minimalistic interface but it captures a small API. I know what I want and we can even discuss parts of the design given these small lines of code.
Of course there's a ton of various things I need to add - some event loop facility, networking support to be able to connect remote computers, reliance of the message queue, potentially other behaviours, but I could very well code up something that would work quite OK in process just given this simple test. Almost like magic!
Never underestimate the real problem in distributed computing
It could easily be the case that we actually would be in need of quite a different solution than the one we thought we needed - maybe we didn't parallelize the problem enough and did all the socket shuffling in vain (I've seen that more than once). And how do we handle that connections are lost? Can we unsubscribe a bus? What happens if we subscribe to many queues. What if servers are down? New servers join the cluster? Authentication? Am I the first person in the Universe facing this particular problem and is it unique of its kind? (that last one was a rhetoric question).
Depending on the specific problem the computing cluster should solve, there are many different ways to approach it. My small TDD approach has the insidious side-effect that it actually makes me narrow down my view of the problem way too early. It's very hard to kill code you wrote start over, almost from scratch. Good test takes effort to specify (and rightly so). To refactor tests probably takes even more effort than to refactor code.
The TDD way to navigate through the large space of solutions, can easily get stuck on a local maxima.
I can do nothing but approve Rich Hickeys idea of "hammock time". You really have to understand and be able to keep the whole problem you try to solve, and as many of its parameters and quirks. One good way to do this is really to try to solve the problem (preferably on paper) and after that see how others did solve similar problems, code up a small prototype, see how others code would solve the same problem, make sure you know how to the whole stack would work. When this work is done: TDD would work just fine.
TDD is a great scaffold, but a most often a really shitty sketch. Actually test cases have too high fidelity and is to slow to write and change, and you can mistake them for "real test cases that should be used to test the final solution on".
Test cases is not a good blue print either - test cases does not say how the solution should work, only how it should behave. If you really want to code this way (which is really powerful), use a declarative programming language, like datalog instead of abusing your precious brain time with being a manual compiler for your own, made up, non-standard, most likely hard-to-understand declarative programming language. (Those are more common than you might think).
In summary
The major risk with TDD is that one get carried away on the wrong track, starts to solve some problem, get some kick from passing tests, continues in that direction for more passing tests kicks, and never really takes the opportunity to really think and reason systematically about the whole problem one have at hand.
Don't be mean to yourself and your friends, always think hard on your problem and possible solutions AFK before coding. If that is not possible, something is wrong, and your code will likely be as messy as the problem it tries to solve.
When you know exactly what the outlines of the problem are and how you want to solve this problem - then TDD is one of several tools to get your code super duper great. Use it accordingly. Thanks.
Prenumerera på:
Inlägg (Atom)