
The Python Podcast.__init__
Fast, Flexible, and Incremental Task Automation With doit
Dec 27, 2021 · 39 min · Episode 345 · 32.3 MB
0:00-39:27
Streams straight from the publisher. podnod never proxies or re-hosts episode audio.
Summary
Every software project needs a tool for managing the repetitive tasks that are involved in building, running, and deploying the code. Frustrated with the limitations of tools like Make, Scons, and others Eduardo Schettino created doit to handle task automation in his own work and released it as open source. In this episode he shares the story behind the project, how it is implemented under the hood, and how you can start using it in your own projects to save you time and effort.
Announcements
- Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Your host as usual is Tobias Macey and today I’m interviewing Eduardo Schettino about Doit, a flexible and low overhead task automation tool
Interview
- Introductions
- How did you get introduced to Python?
- Can you describe what doit is and the story behind it?
- What are the main goals and use cases of doit?
- Can you describe how you approached the implementation of Doit?
- How has the design changed or evolved since you first began working on it?
- The realm of task automation tools for developers is an exceedingly crowded one, with each tool prioritizing certain use cases. How would you characterize the position of doit in the current ecosystem?
- How does it compare to e.g. Click, Invoke, Typer, etc.?
- What is your guiding philosophy for when and how to add new features?
- You have been running the project for ~13 years now. How has the evolution of the Python language and ecosystem influenced your approach to the development and maintenance of doit?
- What is the workflow for getting started with doit and integrating it into your development process?
- For every project there are some tasks that are identical and some that are bespoke for that application. What are the options for maintaining a standard set of tasks across repositories and composing them with per-project activites?
- What are some of the useful patterns that you and the community have established for designing tasks and execution graphs?
- How do you use doit in your own work?
- What are the most interesting, innovative, or unexpected ways that you have seen doit used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on doit?
- When is doit the wrong choice?
- What do you have planned for the future of doit?
Keep In Touch
- schettino72 on GitHub
Picks
- Tobias
- The Matrix series
- Eduardo
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
- doit
- Zope
- Twisted
- Django
- Pyflakes
- scons
- Make
- Nikola
- Nose
- Pytest
- Click
- Typer
- Invoke
- Puppet
- Ansible
- Chef
- Sphinx
- Snakemake
- Airflow
- Luigi
- pytest-incremental
- import-deps
- dbm
- MetalK8s
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
pythonpodcast.com/linode
pythonpodcast.comLinkedIn
linkedin.comschettino72
github.comThe Matrix
en.wikipedia.orgJohn Pilger
johnpilger.comData Engineering Podcast
dataengineeringpodcast.comsite
pythonpodcast.comiTunes
itunes.apple.comdoit
pydoit.orgZope
zope.orgTwisted
twistedmatrix.comDjango
djangoproject.comPyflakes
github.comscons
scons.orgMake
en.wikipedia.orgNikola
getnikola.comPodcast Episode
pythonpodcast.comNose
github.comPytest
docs.pytest.orgPodcast Episode
pythonpodcast.comClick
click.palletsprojects.comTyper
typer.tiangolo.comInvoke
pyinvoke.orgPuppet
puppet.comAnsible
ansible.comChef
chef.ioSphinx
sphinx-doc.orgSnakemake
snakemake.readthedocs.ioAirflow
airflow.apache.orgLuigi
luigi.readthedocs.iopytest-incremental
pytest-incremental.readthedocs.ioimport-deps
github.comdbm
docs.python.orgMetalK8s
github.comThe Freak Fandango Orchestra
freemusicarchive.orgCC BY-SA
creativecommons.org
- 0:12Introduction and Sponsor Messages
- 1:36Interview with Eduardo Scattino Begins
- 1:55Eduardo's Background and Introduction to Python
- 2:33The Origin and Motivation Behind DoIT
- 4:15Main Goals and Use Cases of DoIT
- 6:48Design Elements and Architecture of DoIT
- 10:01Comparison with Other Task Automation Tools
- 16:09Community-Driven Development and Key Features
- 18:03Impact of Python Ecosystem Evolution
- 20:37Getting Started with DoIT
- 23:04Dynamic Task Graph Building
- 24:50Reusable Task Flows
- 26:00Discovering and Extending Tasks
- 28:10Debugging and Testing Tasks
- 29:19Eduardo's Personal Use and Interesting Projects
- 34:11Interesting Uses and Projects Built on DoIT
- 35:03Challenges and Lessons Learned
- 35:59When Not to Use DoIT
- 36:38Future Plans and Contributions
- 37:29Closing Remarks and Picks