Ep 107: Idempotify!

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to feedback@clojuredesign.club, or join the #clojuredesign-podcast channel on the Clojurians Slack. This week, the topic is: "handling endless errors". We discover when giving up is the way to get ahead. Our discussion includes: Finishing up our Sportify tracer-bullet implementation. What context can you put in a file name? Tricks on working with time information. Trying to "fix" a situation vs adding more context. What is a "positive representation" of a problem? What are "fundamental" versus "derived" facts? What information tends to be more stable? What information should you try to co-locate? How do you handle resources that are not currently available? How many times should you retry? How long should you wait between retries? How do you figure that out? What is idempotency? How can it help? At what point should you stop trying to handle errors? Can you have too much automation? How can you troubleshoot intermittent problems? Selected quotes We don't need a time zone offset because we know it's in UTC! In the spirit of building up the language to meet our domain, we can write a pure function! We want a deterministic way to go from this kind of common information into all the other bits of derived information. I was really hoping that we would finally be done with the errors and we could just get a highlight clip, but yet again, the world has conspired against us to make our life difficult as a programmer! Hold on! Hold on! The first thing we should do is run our process again because maybe the error will just go away! The problem does not go away in this instance. The problem just goes back to being hidden! The frustrating thing about programming is that code will do exactly what you told it to do. Clojure is already positive about nothing, that's what we call nil, so why not be positive about bad stuff too? Now I'm personifying the function as myself! It's better to reference information by its main identifier as opposed to some derived identifier. The context is all over the place: some context in memory, some context in the file system, and some intermediate context in the imperative function. That's the last problem we encounter, right? You never know what the world's going to throw at you! Why don't we just run it again? Let's just run it again! Maybe it'll be ready now... Maybe now... Maybe it will be ready now... And then we hit another error which is: the MAM rejects us for making too many requests! How long do you wait? Should you back off? That adds a lot of complexity to the code at that level. Because adding idempotency increases the complexity of that part of the solution, we only want to add it where it's necessary. The error condition is happening right now, so let's write the code to fix it right now. That's the way automation is at some point in time: a human needs to do it. You can find yourself in a situation where you're trying to do too many automatic things. Is it really worth doing? You cannot solve every possible permutation once you're interacting with the real world. It's okay to put up the guardrails, and if I'm outside them, robot me throws up my hands and says, "Human please!" But what happens when you run it again and the clip is ready? Now your retry logic doesn't get tested. Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Related episodes: Ep 027: Collected Context Ep 029: Problem Unknown: Log Lines