Clojure ‘def’, ‘declare’, and ‘concat’

I hit the Aeron hard and asked my pair Liz “what’s up?” I’d been sick and she had been working on a tricky Datomic datalog query in my absence. She claimed to have solved the whole problem but the tests would not pass when run as a whole vs. when run individually.

A classic problem with a big Clojure twist. I, being no stranger to this terribleness, eventually shook the sick out of my head and asked if she had tried running the singular test file by itself. Good news: it failed too.  What? I’ll explain.  We use a classic trick here at Backstop when running tests in the Clojure REPL: ‘:reload-all’

(require '[clojure.test :refer [run-tests]])
(require 'your-ns.example-test)
(run-tests ‘your-ns.example-test) ; First time

(require 'your-ns.example-test :reload-all) ; Picks up new changes in the files
(run-tests ‘your-ns.example-test) 












Otherwise we’d spend all day waiting for the JVM to start up every time we run our tests. It’s a cool trick but it has its limitations. We were clearly dealing with one of those limitations as the tests would fail once in a fresh REPL but succeed on a second run.  Those who’ve already guessed the answer please apply for a job here. For all us other mere mortals who need to think it through I’ll present this example of what we were trying to do and what happened:






(declare a-vector)
#'user/a-vector
(def time-bomb (concat [1 2 3] a-vector))
#'user/time-bomb
(def a-vector [4])
#'user/a-vector
time-bomb
IllegalArgumentException Don't know how to create ISeq from: 


clojure.lang.Var$Unbound  clojure.lang.RT.seqFrom (RT.java:505)












But if you asked for time-bomb again:






time-bomb
(1 2 3)








Everything is, er, fine?!?  Except 'a-vector' has disappeared and a 'function' has returned different things for the same lack of input.  So, of course, “concat” is part of the problem here. As Stuart Sierra pointed out in http://stuartsierra.com/2015/04/26/clojure-donts-concat,  “concat” is a lazily evaluated join so it can hide many bombs. 












If we had instead used “into,” which is not lazily evaluated, all would have blown up in a nice normal manner:






(declare b-vector)
#'user/b-vector
(def now-bomb (into [1 2 3] b-vector))
CompilerException java.lang.IllegalArgumentException: Don't know how to create 


 ISeq from: clojure.lang.Var$Unbound,


 compiling:(form-init1552771854893412833.clj:1:15)








Well, not a great error but at least the line number would have given us a clue as to where the problem was instead of the stack trace starting where ever the “time-bomb” was evaluated. 












However, “a-vector” is defined by the time “time-bomb” is evaluated. So why doesn’t the lazy “concat” use the available “a-vector”? “Concat” seems to freeze the temporary nature of clojure.lang.Var$Unbound and not evaluate a later defined “a-vector.”  Weird. Is this a bug?  Or just a known evaluation order thing? I verified this behavior in 1.5,1.6,1.7, and 1.8 so it seems here to stay.












In case you are wondering, without a “declare” this code would blow up right away on the right line number:






(def nope (concat [1 2 3] nothing-at-all))
CompilerException java.lang.RuntimeException: 


  Unable to resolve symbol: nothing-at-all in this context, 


  compiling:(/tmp/form-init1552771854893412833.clj:1:11)


















Why did we use declare instead of just order the code so it compiles? As a team, we’ve adopted a code standard where we try to push private functions down to the bottom of a file and “declare” helps us declare things that will be coming to the compiler/jvm/macro-magic-factory. It works really well with “defn” so we applied similar principles to public “def”s (for use outside the namespace) and “def”s we didn’t think should be part of a namespace’s API.









The combination of testing in the REPL, code conventions, the lazy nature of “concat,” and when “def” gets evaluated created a lot of confusion for us. Hopefully, this blog post helps you figure out some similar problems.


































Comments
















David Chelimsky said…




While you've definitely hit an odd gotcha, I'd strongly recommend you reconsider using declare as a means of reorganizing functions in a namespace. Its real use case is to support circular dependencies between functions, but it comes at a cost. You've already witnessed one problem. Here's another one you might see down the road: you declare some-var, reference it in a sadly untested conditional branch, everything compiles, all tests pass, deploy and ... kablam! In production! I'm sure you're thinking "but that will never happen to us because TDD or because peer code review or etc, etc, but why expose yourself to this risk? Also, your team is now engaging in a practice that most Clojure projects don't employ, so your code will look foreign to the next experienced Clojure dev you hire, and it will make your devs sad when they go to other teams that don't follow this convention. That's my 2 cents - for you it's free!






February 1, 2016 at 7:42 PM
























Tyler Jennings said…




You found a real gem here!  It very much seems like a bug to me.  I can't think of any reason this could be appropriate behavior.






February 15, 2016 at 11:52 AM




















Post a Comment
























Popular posts from this blog









What's a Good Flog Score?





















I've been using Flog  to measure the complexity of my Ruby code and I was wondering how others interpret Flog numbers.   I'll go first with my opinions: Score of    Means 0-10        Awesome 11-20       Good enough 21-40       Might need refactoring 41-60       Possible to justify 61-100      Danger 100-200     Whoop, whoop, whoop 200 +       Someone please think of the children (note: these are scores for an individual method) The 20 - 60 range is interesting to me.  Sometimes the complexity of what you're doing will justify the score and other times the method is in desperate need of refactoring. The highest Flog score I've seen in the wild was in the 300's.  What's you're personal high and what do you think of my rankings? In other news, I've redesigned the look of my blog so if you only see my post through a reader you might want to stop on by, take a look, and tell me what you think.








Read more










SICP Wasn’t Written for You





















The number of software luminaries who sing the praises of “ Structure and Interpretation of Computer Programs ” (referred to as SICP) is such a long list that you might think only a crazy person would take issue with it. However, to ignore SICP’s problems and continue to blindly recommend it seems just as crazy. SICP was the textbook for MIT’s introductory programming class and was a bit of a departure from other into to computer science textbooks at the time.  Wikipedia sums it up nicely:  “Before SICP, the introductory courses were almost always filled with learning the details of some programming language, while SICP focuses on finding general patterns from specific problems and building software tools that embody each pattern.”  Which sounds awesome, but does essentially say that abstract principles will be introduced before the nuts and bolts of a language.  If you think about that for a minute, you may see where the problems will be. When I was training to be a...








Read more










Point Inside a Polygon in Ruby





















Recently my team needed to find out if a point was inside a given polygon and, as usual, we found some code on the internet  that did what we wanted.  And, as usual, there wasn't much explanation as to what it was doing.  Here's the code (after we converted it to Ruby):   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17   def  contains_point? (point)   c = false   i = -1   j = self .size - 1   while  (i += 1 ) < self .size     if  (( self [i].y <= point.y && point.y < self [j].y) ||        ( self [j].y <= point.y && point.y < self [i].y))       if  (point.x < ( self [j].x - self [i].x) * (point.y - self [i].y) /                     ( self [j].y - self [i].y) + self [i].x)         c = !c         end       end       j = i     end     return  c   end end So, you know, it works and I'm grateful to somebody for posting it but we had no idea how it works.  So we wrapped 2 tests around it (one where the point is not in the polygon and another where it is...








Read more