Jake Scruggs

writes ruby/wears crazy shirts

I hit the Aeron hard and asked my pair Liz “what’s up?” I’d been sick and she had been working on a tricky Datomic datalog query in my absence. She claimed to have solved the whole problem but the tests would not pass when run as a whole vs. when run individually.

A classic problem with a big Clojure twist. I, being no stranger to this terribleness, eventually shook the sick out of my head and asked if she had tried running the singular test file by itself. Good news: it failed too.  What? I’ll explain.  We use a classic trick here at Backstop when running tests in the Clojure REPL: ‘:reload-all’

(require '[clojure.test :refer [run-tests]])
(require 'your-ns.example-test)
(run-tests ‘your-ns.example-test) ; First time

(require 'your-ns.example-test :reload-all) ; Picks up new changes in the files
(run-tests ‘your-ns.example-test) 

Otherwise we’d spend all day waiting for the JVM to start up every time we run our tests. It’s a cool trick but it has its limitations. We were clearly dealing with one of those limitations as the tests would fail once in a fresh REPL but succeed on a second run.  Those who’ve already guessed the answer please apply for a job here. For all us other mere mortals who need to think it through I’ll present this example of what we were trying to do and what happened:

(declare a-vector)
(def time-bomb (concat [1 2 3] a-vector))
(def a-vector [4])
IllegalArgumentException Don't know how to create ISeq from: 
clojure.lang.Var$Unbound  clojure.lang.RT.seqFrom (RT.java:505)

But if you asked for time-bomb again:

(1 2 3)

Everything is, er, fine?!?  Except 'a-vector' has disappeared and a 'function' has returned different things for the same lack of input.  So, of course, “concat” is part of the problem here. As Stuart Sierra pointed out in http://stuartsierra.com/2015/04/26/clojure-donts-concat,  “concat” is a lazily evaluated join so it can hide many bombs. 

If we had instead used “into,” which is not lazily evaluated, all would have blown up in a nice normal manner:

(declare b-vector)
(def now-bomb (into [1 2 3] b-vector))
CompilerException java.lang.IllegalArgumentException: Don't know how to create 
 ISeq from: clojure.lang.Var$Unbound,

Well, not a great error but at least the line number would have given us a clue as to where the problem was instead of the stack trace starting where ever the “time-bomb” was evaluated. 

However, “a-vector” is defined by the time “time-bomb” is evaluated. So why doesn’t the lazy “concat” use the available “a-vector”? “Concat” seems to freeze the temporary nature of clojure.lang.Var$Unbound and not evaluate a later defined “a-vector.”  Weird. Is this a bug?  Or just a known evaluation order thing? I verified this behavior in 1.5,1.6,1.7, and 1.8 so it seems here to stay.

In case you are wondering, without a “declare” this code would blow up right away on the right line number:

(def nope (concat [1 2 3] nothing-at-all))
CompilerException java.lang.RuntimeException: 
  Unable to resolve symbol: nothing-at-all in this context, 

Why did we use declare instead of just order the code so it compiles? As a team, we’ve adopted a code standard where we try to push private functions down to the bottom of a file and “declare” helps us declare things that will be coming to the compiler/jvm/macro-magic-factory. It works really well with “defn” so we applied similar principles to public “def”s (for use outside the namespace) and “def”s we didn’t think should be part of a namespace’s API.

The combination of testing in the REPL, code conventions, the lazy nature of “concat,” and when “def” gets evaluated created a lot of confusion for us. Hopefully, this blog post helps you figure out some similar problems.

Landing in a pile on a large stone table in a massive room from out of a swilling green magic portal is very disorienting. However, it did lead to Lutz shaking off his magical blindness. First sight: Many intense dudes some covered in lacquer armor — Lutz didn’t even know lacquer was a thing you could make armor from. While Lutz pondered the truth of his eyesight, Lord Valparaiso threw a little fit. One: He was mighty pissed that his evil portal trap has ensnared him in addition to our heroes. Two: The portal had clearly been hijacked to another mysterious and far away land. The land was so strange and far from their home that Raest couldn’t feel his god Palore anymore which meant no spells.  Essentially this changed his role in the party from from healer/crappy fighter to, um, crappy fighter. That single set of footprints on the beach are not from Palore carrying Raest during his darkest hour. 

The team had many questions but lacking a common language they were ignored, shoved out of the big room, down a hallway, and into a smaller room with a chalkboard. A very painful session of explanation ensued. Drawings of stick figures and angry gestures soon left both sides exhausted. The time had come, once again, for Lutz to put the team on his back. Working with noted jerky-owner and consumer Bran they managed to extract salt from beef jerky infused water with heat from a torch. That pinch of salt combined with a pinch of soot plus Lutz’s NEW FOUND ARCANE TRICKSTER POWERS meant he could cast a Comprehend Languages spell. Clutch. 

Now that they could understand what was being said, via Lutz, the strangers in a strange land understood that they must play a spirited game of foots-ball with big bats and a medicine ball. There would also be giant walking trees that shoot hurt-y needles if you get too close. Hitting other players was allowed and encouraged but the ultimate goal was to get the 2 foot wide medicine ball into the goal by any means necessary. Winner gets not to die. Best 2 out of 3. Classy.

Our ambassador/coach/warden gave the team a (perhaps) last meal. It kinda seemed she might have an interest in seeing team Lutz win. Was betting involved? Who’s to say? Mind your own business. The important part of this last feast was that everyone got a rest and Lutz hid away a small amount of alcohol. 

Upon reaching the field for the first match the team saw their competitors. From far away they appeared to be a bunch of imps and a man.  As they came closer the dread increased as our brave band of adventurers saw one 9 foot tall center and a bunch of good sized dudes. 

At the start of the match more bad news came pretty fast: Clustering around the ball made the team susceptible to a sleep spell from the safely 50 foot back magic user. The other team had “forwards” running down the wings, defenders hanging back, and an actual strategy. Peren, Lutz, Bran, and Raytheon had more of a “swarming bees” strategy that kinda played right into team bad guys’ “cast a sleep spell and send a man down the side lines” strategy.

You might think the gang got out of this one with some clever thinking, but no, they mostly just whacked everything nearby which included the ball. Ray-ray, in particular, took out his frustrations over losing all of his promising new cult by smashing every opponent excessively: Often from behind. Somehow this aggressive flailing got the ball past the team jerk face’s front line and rolling. The big bad, being not very big on smarts, crushed the ball towards the wall which was about 90 degrees off from his target goal. Lutz used his superior speed and agility to do 2 things in the time another would merely perform one. He wake up the sleeping members of his team and flanked the opposition. Bran got all “angry barbarian” and made like Lutz with the 2 to 1 actions. Unfortunately the other team had a double attacker too - lots of hustle displayed on the pitch that day. The first match was won by Lutz picking up the ball, using his crazy moves to break the ankles of the other team, and bursting on through to the goal.

During the break Lutz cast a False Life upon himself with his fly arcane trickster powers using the dram of alcohol he had acquired from the “last supper.” Palore, wherever he may be, did not favor Raest with the heal spell he so desperately needed. Only a jerk would have pointed out that necromancy had succeeded where religion had failed. A bit more rested than before the team braced for the coming match. How would the other team solve the “Lutz problem?”

Well how could you? He’s just too damn fast and agile.  Like they used to say about famed foots-ball player Mycroft Jordan-Hammer: You can’t stop a Lutz, you can only hope to contain him. Once again Lutz woke up sleepy members of the team, picked up the ball, and made a break for it. Problem was that this time a terror tree decided to get between Lutz and the goal. Now a normal thief, pick pocket, 2nd story man, or even assassin would perhaps have turned tail at that point. Lutz sucked up his courage, cast “Thunder Wave,” and knocked the terror tree aside. An entire walking tree pushed back 10 feet by a skinny studded leather wearing punk of an 18 year old. Just picture it. Now picture a bunch of jerks chasing Lutz for a few long seconds as it is became apparent to all that it was going to be a 2-0 sweep. Clutch.

Now that they won the right not be slaughtered for a god’s favor, how will these heroes be treated in a land even Palore has forgotten? Tune in next time to find out.

Recently I was interviewing a candidate for a job on my team at BackstopSolutions and he said something like: “I read your blog post about Microservices and was wondering if the lack of follow up meant the project was a failure.” He actually expressed this sentiment in a much nicer way as this was an interview but still it got me thinking: "I should do an update on the status of our microservice and explain how it could be purely functional."

The service is implemented in Clojure, uses Datomic for persistence, and integrates via REST.

The service has been in production for several months now and has gone through a number of versions of its API. Implementing HATEAOS worked out well as it allows us decouple having to keep the service and front end in lock step. Since this is a microservice, we could keep both applications in step with each other as we own both ends of the interface. However this co-ordination is not without cost.  Release dates slip b/c of bugs in either system.  There are less bottlenecks in production releases as the front and back end can be deployed to production independently of each other. Not having to co-ordinate releases is a wondrous decoupling. It didn’t take long to implement HATEAOS and maintenance costs for the discoverability aspect have not materialized.

Datomic is a special snowflake of a database and that it why I believe it is the future. That is a bold statement deserving of a whole other blog post so I will merely list a few of Datomic’s features that we exploit:
  1. It never forgets. Think of it like source control but with transactions replacing commits.
  2. Time is a first class citizen. This is really an outgrowth of the above feature but is worth pointing out on its own.
  3. It uses the incredibly compose-able Datalog query language instead of SQL and can include clojure.core expressions in its queries.
  4. It can easily be used as a graph database.

Those are some crazy advantages. Since Datomic never forgets, ANY query to our service that specifies a time can never have a different return value. That is what I mean when I say that our service is “purely functional.”  This enables the Rails app that works with our micro service to cache every query it makes FOREVER. The service without caching is still crazy fast considering the incredibly complex queries is has to run. Like any graph database, speed is determined by what percentage of the graph a node can access. Even if a customer stretches the engine to its maximum, our ability to cache forever means they will take that hit but once. Of course you can query our microservice with no time specified where it will default to now and not cache. This is used only for debugging and is never used by our Rails application.  

The graph database aspect of Datomic combined with its understanding of clojure.core enable us to spend days instead of months (or even years) implementing custom business logic. Backstop Solutions is a Business to Business (B2B) Software As A Service (SAAS) firm.  In B2B there is but one constant:  All business’ processes are unique. Period: End of story. 

There may be some similarities and generalities between companies in the same competitive space but these similarities are revealed to be mostly illusionary when you try to implement specific customers. Drill down into any business’ rules and you will find unique requests from every client.  Think of it this way: In an efficient market what use is there for complete duplication? The clients who's processes are a competitive advantage hate any SAAS offering that can not implement aspects of their system in the SAAS product. With this microservice we can. The actual challenge is to get clients to simplify their rules enough so that they themselves can understand the rules. This may seem confusing until you consider that Backstop Solutions has clients with many offices in many different lands so we are often the first people to implement a logically consistent system that fits all of their needs, people, and processes.

Datalog is a subset of Prolog. Prolog is a logic based language as is SQL. Where it crushes SQL like a bug is its compose-ability. You can create a bunch of “rules” and mix and match these rules just like you are used to composing functions. This is amazing. I can not stress this enough. I continually implement stories in a fraction of the estimated time because I discover that most of the query is already written and I just have to tack a little on the end, or beginning, or middle. 

What is even more wonderful is that even if I ever get into a tight corner with Datalog I can fall back on all the functions in clojure.core.  How many times has it infuriated you that the version of SQL you are using can’t support a completely logical function and so you have to do something horrible? Now imagine being able to use all of clojure.core in SQL. Exactly.

I should stop and point out that I had the idea for this service 4 years ago during a conversation with David Chelimsky. As he described a problem he was solving with a graph database I realized that we had a completely different but functionally similar problem where I work. Thanks for always being inspiring David.

The “Curious Case of the Tiny Thieves” is being joined in progress and so a little backstory is necessary:

The dexterous and rather handsome rouge Lutz was investigating these “Tiny Thieves” with his band of mostly merry adventurers: Raest a cleric of Palore, Bran the mighty barbarian, Torment the super naive atheist paladin, a wild magic user called Very Forgettable Name, and Iesha or Aida or Ada AKA: the old lady magic user who casts a lot of sleep spells. They discovered the little thieves trying to abscond with their solid gold cat statue which they received as a gift for returning a talking cat to an old lady who was really a dragon in human form — long story.

Turns out the whole town hated these mini scofflaws as they has been on a veritable spree of the crime variety.  A reward was offered and pursued by our fairly fearless heroes. As far as they knew the criminals were: 6-8 inches tall, never seen in full light, and immune to sleep spells. Lutz made a super cool trap for these minute miscreants but they avoided it perhaps because the gang had left the lights on. Seems obvious in retrospect… They did find tiny foot prints in the fireplace the next day!  They were also found the same footprints the next room over where a very expensive necklace had been stolen. Tiny thieves: 1. Adventurers: 0.

That was all prologue, believe it or not.

Upon investigating the roof and dirt surrounding the “Sleepy Dragon” the team supreme found evidence of rope used to climb down the chimney and footprints leading away from the inn.  Following the trail of regular sized footprints the gang pretty much got lost right away.  Sigh.  The trail was definitely headed south and that’s about all they knew. In that direction lay:  The bad part of town, The duke who had suspiciously never been robbed, and a new unknown church that ministered to the poor in said bad party of town. “Why not stop in at the working class church and ask around?” thought pretty much everyone.  Raest, who apparently has more money than sense, decided that 5 gold was an appropriate bribe/offering to a dirt poor chump. Not 5 copper, mind you, but 5 gold. As in 500 copper.  Bran had been invited in to the small place of worship by Raest in a move that was so very sensible it belied his previous insane tipping/bribing strategy. I feel like I’m focusing on this too much… But seriously 5 gold. FIVE DAMN GOLD. It makes a rouge wretch to think of it.

Iesha noticed the lay priest was totally lying about everything with her wicked awesome old lady perception skilz. However that information was disclosed after Raest and Bran had disappeared into the hovel/church.

Lucky Lutz, the super awesome leader of the group, had wisely decided to keep his ear to the door and therefore heard screams from within. First through the door and down the stairs Lutz  sprinted only to be held up by the barbarian’s massive size in the stairwell. Bran had been stopped by…

The Tiny Thieves! They were clearly automatons (explaining why they could not be put to sleep)  and surprisingly strong as they managed to subdue the brick shit house that is Bran and even Lutz: the slipperiest rouge in the land. Unfortunately for those wee jerks, Torment has a bewildering talent with his net. He effortlessly swept them off of Lutz from 10 feet away!
But what of Raest and his terrible ability to estimate bribes?  He was in another corner of the savage basement of heck getting all perforated by the lay priest of a bad guy god. That’s right: Raest had paid 5 gp for an invitation to an underground murder party.

Raest cast a quick command spell which, as you may know, can be only one word. That word was “retch.”  Cult-y McJerk Face then heaved up his lunch while getting pulverized via mace to the face. He died fast in a bewildering flurry of vomit and steel. Despite this early success Raest was lit up like a Christmas tree by a bunch of magic bolts from a pair of magic jerks.  Torment barely missed falling into a pit trap and then charged the head cult guy who's name probably starts with a V. Sadly V(icky?) introduced Torment to the Xmas tree experience mortally wounding the paladin. Torment collapsed to the ground unconscious and bleeding out. Bran and Raest kicked ass for awhile but succumbed to the combined forces of the tiny thieves and dickhead magic users. Low on health, they were put into a magical sleep and were soon to be easy pickings. Surely Lutz could have saved the day had he not been magically blinded and therefore unable to help his friend Torment dying just a few feet away. Aida was upstairs drinking water and that other guy was investigating the latrine. Really. They may or may not have had a slap fight while getting down the stairs.

Things looked dire for our rag tag group of misfits until the cult leader made a mistake. Which was good because the only thing stopping his dance of diabolical destruction was the late arriving, but very hydrated, team of Aida and wild magic guy. Did V(oldemort?) over estimate the strength of his magic portal’s vacuum power? Or did he simply slip like the jackass he was and always will be? Maybe we’ll find out next time because everyone but the old bat and a mostly dead Torment got sucked through the portal into another land.

Another land of suck. 5 seconds of being in the new place revealed 2 things:
1) The new bad guys are plentiful and way into lacquer armor.
2) Lutz and company are to be offerings of some sort. Which doesn’t sound good at all.


Hearing that David Heinemeier Hansson had slammed microservices in his Rails Conf 2015 keynote was a bit disorienting as I was developing a microservice at the time. I leaned over to the other senior developer and said: “What’s his deal?” Toby Tripp, who had just watched the keynote, explained that when David’s team implemented microservices they had terrible trouble managing all the versions between 3 systems. “Isn’t that why we use a lookup call to get the URI-map?” I asked.
“Yep” he said.
“That and never remove a return key/value (pair), only add new ones until you can prove they are not used anymore” He added.
I returned to adding in some caching to our service calls. 

Later, I watched the video which is here:

David has a meandering style but he mostly discusses microservices between 43:00 and 52:00.  The “what’s in your backpack” stuff is all about his earlier zombie apocalypse metaphor that you can watch if you like.  He claims his team implemented microservices right, loved them on day 5 but hated ‘em in year 2 (paraphrase). I don’t understand how I will regret my microservice architecture decision if we implement HATEOAS (Hypermedia as the Engine of Application State).  You may remember HATEOAS as the thing from REST that all the eggheads complained was missing from Rails’ “RESTful-routes.” “Sure, the routes were REST but what about the discoverable URI-map?” they may have said.  They were, of course, right (jerks).

I’m not saying HATEOAS is perfect.  I was in the middle of adding LOTS of caching to our client so it almost never has to call our service for one of the two calls you must, at a minimum, make with a HATEOAS service. Why two calls? 
  1. To get the URI-map (A map of how to call the service for all resources)
  2. Actually make the call (Using the above map)
  3. What? There’s a 3? Yup, that second call could return another URI-map
  4. It could continue

I know this seems insane, to make 2 or more http calls for every one request, but the benefits outweigh the costs. Old calls will always work. Less bugs. More confident software.  Also, you can cache the hell out of all the URI map requests from the client side.  When you need to expire that cache you can do it upon release of new functionality because, this being a microservice, you own all sides of the interface. Just work it into your deploy script and you’re done. Extra bonus: No URI ever has a stupid v7 in it.  

The last call is a bit of a bummer over http. Even servers right next to each other in the cage have important amounts of latency for many services.  HATEOAS isn’t for every service. Although, with virtual machines, I run my service on the same physical box in another vm: Somewhat mitigating network latency.  Double Extra Bonus:  My service is purely functional so I can cache all of its responses forever. That bold claim, however, is another blog post.

Anyway, Tripp is leaving my team to go be the king of software at some company I refuse to promote out of spite. How would you like to work at Backstop Solutions? I’m looking for a senior Ruby dev with a bunch of Clojure experience but, to be honest, I don’t really expect to find one. So if you’ve got some decent Ruby skills and can learn Clojure while pair programming you should apply.  https://www.backstopsolutions.com/careers

Mention you read this article in the interview and you’ll get to see a man in a crazy shirt blush.

Recently I re-tweeted this thought from Pete Holmes:

because I'm in the middle of a communication crisis.  I do "knowledge work" so it’s considered inappropriate to have a device constantly making little beeps and boops while the person next to me is working on some insane LibXML (you don’t wanna know) bug.  Programming requires extreme concentration and distractions are to be avoided. 

The rub of it is that I’m “on call” if our software product bursts into flames.  Therefore, I must keep my phone in a mode capable of disturbing me so I don’t eat lunch, play ping-pong, or just code right through the disastrous failure of our software. 

This used to be fine back when text messages were either:
  • Time sensitive
  • Important
  • From one person
In first two cases the beeps my phone made were appropriate and in the last case I could simply text back to that person that I was busy if they didn’t get the me ignoring them hint.  However, as we all sadly know, the plague of group texts has completely disrupted how we handle notifications.  Now when my mother in-law, a nice person who likes to share, sends me and 10 other contacts a pic of a lake I spend the rest of the afternoon getting “Nice!,” “Did you take a swim?,” “No, it’s too cold”, “where are you guys?” etc. texts.  Every one of those I should probably check to make sure it isn’t an important work thing — just in case. 

Why not a mass email? Why do I get group texts trying to plan something days or weeks away? Shouldn’t that be an email thread?  The answer is as obvious as it is sad-making: People have given up on email.  For every Inbox Zero zealot, there are a hundred people who’ve essentially let their inbox run wild with clutter.  Missed important emails because of that clutter? — better move to text messages For Everything.  Of course, this leads to cluttered text messages. Plenty of times I’ve been furiously searching through my email to find someone’s reply only to remember “Oh, they’re one of those text-y people, I better scroll through all the various group texts threads they’ve been involved in… Sigh.”

A small aside:  Some people don’t have unlimited text messages on their phone.  The horror, right?  My boss consistently crashes through her text message limit b/c of “friendly” group messages.

The problem of communication clutter isn’t going away.  Ever. Abandoning email for texts just moves the problem.  I implore you to take charge of your inbox, people.  Getting emails you never read? Unsubscribe or Block! Can’t bring yourself to do that? Well, how about creating a rule that moves your daily…

Hold on, got a text…  yup, a silly friendly chatty one from a friend who never emails anymore.  I really like this guy but such frivolity is a tweet or an email or a Facebook or a Whatever but not a text that demands my attention.

Where was I…  Oh yeah, create rules in your email client of choice that automatically move “less often read” messages to a folder for later (read: never) reading. If you’re still getting a flood then you must keep going. Unsubscribing, blocking, and rules are your new watchwords. If you can’t handle this fire hose of information now what do you think is going happen when everyone can send “animated emojis” on a whim from their Apple Watch (or some other wearable computer thing)?

Your “inbox” is not just your email inbox.  It is Every Damn Message you receive in Any form. They all via for your attention and giving up on any one communication form only annoys your friends while providing, at best, temporary relief.

Apology:  I’m sorry if you’ve sent me a group text in the past and see this as an attack.  I really do want to hear about your cats, see your kid’s pics, and ponder your stray thoughts… In an appropriate medium.  Does that sound bitchy? 

Episode 4 “Time to Exercise!” is out right now! Search for it on your favorite podcast app or check out our free temp website here: http://softwareapprenticeship.libsyn.com

With 4 weeks under his belt (plus 9 weeks of Dev Bootcamp) our apprentice, Jonathan Howden, continues his quest to become an enterprise software developer at an amazingly rapid pace.  Can a dedicated man become a good developer without a college degree?  Tune in and find out (spoiler: he’s doing well but it’s intense) 

Topics this week:

  • Doing push-ups to break up the lethargy of coding
  • Migrating from Authlogic to Devise/Warden and the perils of using a framework’s column in the database for activation.
  • Why senior programmers avoid becoming mentors
  • Rails’ Asset Pipeline  
  • The usual screwing around and one censored F-bomb (sorry - it was me).

Yesterday we all sat in a room and reviewed Jon’s chess code (his outside of work coding project).  I’ll try to put up a more detailed article about it soon, but in brief it went well.  It’s fun to watch a junior developer try to encode all the crazy logic of chess while keeping the code clean, tested, and understandable.  Other than making the classic mistake of mocking/stubbing the very object he was testing, Jon has some pretty readable code that will “mostly” let 2 people play chess against each other.

Episode 5’s in the can (on the SSD?) and we’re recording episode 6 later today with Dave Hoover, co-founder of Dev Bootcamp.  Jonathan attended Chicago’s DBC in September of 2013 so Dave will get to check in with how Jon is doing in the “wild.”  Also, Dave and I worked at both ThoughtWorks and Obtiva together so it should be quite an interesting conversation.

I really wanted this to be a weekly podcast but here we are at the end of the 9th week and only recording our 6th ep.  Now that we’ve been through the recording and editing  process a few times it should be easier to stick to a weekly schedule.  Even if we weren’t recording the conversation I would still have a weekly wrap up with members of the team and the apprentice — it is a very nice way to sum up the week and lessons learned.  It's sort of a weekly, recorded, low stakes retrospective.

The views and opinions expressed here are my own and don’t necessarily represent positions, strategies, or opinions of Backstop Solutions Group.