NewsTech News

Software Engineering Daily

softwareengineeringdaily.com

Software Daily: The world through the lens of software.

Play

Open in

2,221 episodes
Avg 53 min
English

Episodes 2,221

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

#1298
Dec 11, 2019 · 48 min
Continuous Intelligence with Kalyan Ramanathan
Logging provides raw data that can be abstracted into higher level information. Logs are generated at every layer of infrastructure: physical host, virtual machine, container, pod, and Kubernetes cluster. Logs are generated by network proxies, edge servers, and API requests. There is far too much logging information to be read by humans. Log messages need to be refined into statistical metrics that can be put into charts. A high volume of log messages can be used to detect anomalies across a system. If unusual behavior is present in a system, the relevant log messages can be identified and sent to a human operator for them to triage and respond to. Kalyan Ramanathan works at Sumo Logic, a platform for log management and continuous intelligence. Sumo Logic recently published the Continuous Intelligence Report, which is based on a study of over 2000 technology companies. This is a useful data set for anyone who is looking to understand adoption of cloud products and Kubernetes, and it can be found at softwareengineeringdaily.com/sumologic. Kalyan joins the show to discuss log management, continuous intelligence, and the data that Sumo Logic gathered in the Continuous Intelligence Report. Full disclosure: Sumo Logic is a sponsor of Software Engineering Daily.
#1297
Dec 10, 2019 · 1 hr
Remote Work with Philip Thomas
Offices have historically been the place where most knowledge work takes place. An office is a central meeting point for everyone in an organization. Offices allow for high bandwidth, in-person communication. Employees have access to shared resources, such as food, tables, and quiet working space. Offices provide a means of encouraging a common culture within a given workplace. There are also significant downsides to offices–most notably the commute. Employees often spend between one and four hours per day driving to the office. Office work creates a huge restriction for parents with a young child to take care of. Many employees feel more productive when they are working at home, or working from a coffee shop. Remote work is becoming an increasingly popular mode of work for knowledge workers. Remote work has been made possible by increased bandwidth, more powerful computers, and new communications software such as Slack and Zoom. Remote work is a powerful trend that is reshaping how knowledge workers spend their time. It is also changing how companies are structured. A “remote first” culture impacts hiring, human resources, engineering, sales, work-life balance and every other aspect of operations. The downstream impacts of remote work will change the labor market even more thoroughly, and cause us to rethink contracting, equity structures, and the traditional five day workweek. Philip Thomas is the co-founder and CEO of Moonlight Work, a marketplace for software engineers who work on contract projects full-time or part-time. Philip is also the co-author of the Remote Work Encyclopedia, which is a collection of strategies and tactics for knowledge workers and companies that are looking to adapt successfully to the changes that remote work is bringing to the world. Philip joins the show to discuss remote work and his experience building Moonlight. Full disclosure: I am an investor in Moonlight Work.
#1296
Dec 9, 2019 · 47 min
Practical AI with Chris Benson
Machine learning algorithms have existed for decades. But in the last ten years, several advancements in software and hardware have caused dramatic growth in the viability of applications based on machine learning. Smartphones generate large quantities of data about how humans move through the world. Software-as-a-service companies generate data about how these humans interact with businesses. Cheap cloud infrastructure allows for the storage of these high volumes of data. Machine learning frameworks such as Apache Spark, TensorFlow, and PyTorch allow developers to easily train statistical models. These models are deployed back to the smartphones and the software-as-a-service companies, which improves the ability for humans to move through the world and gain utility from their business transactions. And as the humans interact more with their computers, it generates more data, which is used to create better models, and higher consumer utility. The combination of smartphones, cloud computing, machine learning algorithms, and distributed computing frameworks is often referred to as “artificial intelligence.” Chris Benson is the host of the podcast Practical AI, and he joins the show to talk about the modern applications of artificial intelligence, and the stories he is covering on Practical AI. On his podcast, Chris talks about everything within the umbrella of AI, from high level stories to low level implementation details.
#1295
Dec 6, 2019 · 1 hr 2 min
Linkerd Market Strategy with William Morgan
The container orchestration wars ended in 2016 with Kubernetes being the most popular open source tool for deploying and managing infrastructure. Since that time, most large enterprises have been implementing a “platform strategy” based around Kubernetes. A platform strategy is a plan for creating a consistent experience for software engineers working throughout an enterprise. At most companies, a software engineer should be thinking about business logic–whether that logic is related to banking, insurance, oil and gas, or e-commerce. Today, engineers at many enterprises need to think about continuous delivery, application deployment, security policy management, and other deeply technical problems that have nothing to do with the business that they are actually working at. Kubernetes is a foundational open source building block that allows enterprises to base the rest of their infrastructure decisions around. Kubernetes has made it much more viable for enterprises to pursue a platform strategy. With widespread adoption of Kubernetes, there is a business opportunity for companies that can offer other platform solutions that build on top of Kubernetes. A service mesh is one such tool. A service mesh provides networking and security features for all the services in an organization. The service mesh category is a large business opportunity because it sits on the critical path of every network request that goes through an enterprise. It is a potential insertion point for lots of other products including logging agents, distributed tracing, network packet scanning, security policy management, and A/B testing. The potential for business expansion is why so many businesses are entering the service mesh category today, from cloud providers to API gateways. Buoyant was one of the first companies to work on a service mesh tool, with the Linkerd open source project. William Morgan is the CEO of Buoyant, and he returns to the show to discuss the competitive dynamics of the service mesh market.
#1294
Dec 5, 2019 · 1 hr 18 min
Istio Market Strategy with Zack Butcher
Kubernetes has created a widespread system for deploying and managing infrastructure. As Kubernetes has been increasingly adopted, companies are thinking about how to leverage that common layer of infrastructure. With the common infrastructure abstraction of Kubernetes, it becomes easier to adopt other abstractions that are uniform across the entire company. This has created a market opportunity for products such as a service mesh. A service mesh consists of sidecar containers that get deployed alongside services in a distributed system. Each sidecar container is used as a proxy for all the communication that goes through the service it is deployed with. This consistent proxying layer provides each service with benefits such as security, routing, telemetry, and policy management. Istio is a service mesh that was created and open sourced by Google. Istio is built around the Envoy service proxy sidecar and a control plane that manages the Envoy sidecars. Since the launch of Istio, some of the Google employees who were working on Istio have started Tetrate, a company with the goal of commercializing Istio into a product that enterprises will pay for. The market demand for service mesh has been proven, but there are many competitors to Tetrate. Istio is open source and can be commercialized by other companies, as well as cloud providers such as Google and AWS. Linkerd is a service mesh built by the company Buoyant, which was the first company to focus exclusively on this space. There are other companies that are expanding existing products into service mesh: Kong, NGINX, and Hashicorp. Zack Butcher is a founding engineer with Tetrate, and he joins the show to discuss the market for service mesh and the plan for Tetrate to build a business around Istio.
#1293
Dec 4, 2019 · 1 hr
Heroku Infrastructure with Mark Turner
A cloud provider gives a developer low-cost compute infrastructure on-demand. Cloud providers can be divided up into two categories: Layer 1 cloud providers and Layer 2 cloud providers. A Layer 1 cloud provider such as Amazon Web Services owns server hardware and sells compute infrastructure as a commodity. A Layer 2 cloud provider purchases compute infrastructure from a Layer 1 provider and builds a high quality developer experience on top of that compute infrastructure. Heroku was the first Layer 2 cloud provider. Heroku’s first business was to provide a high quality developer experience and low cost containerization infrastructure on top of Amazon’s EC2 virtual machine infrastructure. Heroku has added features for continuous integration, relational databases, caches, and queueing. Building a Layer 2 cloud provider is a very different challenge than building a Layer 1 cloud provider. A Layer 1 provider must focus on low level problems such as hardware infrastructure and virtualization. This does not leave much time for focusing on developer experience. A Layer 1 cloud provider must be able to serve every type of potential software customer. A Layer 2 provider can provide a streamlined experience. Mark Turner is an engineer at Heroku. He joins the show to discuss the architecture and engineering of a Layer 2 cloud provider. Heroku is built on top of Amazon Web Services, and the core compute infrastructure is built on top of a pool of EC2 virtual machines that are continually scheduled with applications that users create on Heroku. Full disclosure: Heroku is a sponsor of Software Engineering Daily.
#1292
Dec 3, 2019 · 45 min
Java 13 with Georges Saab
Java has been popular since the 90s, when it started to be used as a programming language for enterprises. Today, Java is still widely deployed, but the infrastructure environment is dramatically different. Java is often deployed to containers in the cloud. If those containers can share resources, then those containers can share the same underlying Java infrastructure. Java 13 is the most recent public release of Java. The new features in Java 13 reflect the changing demands of modern application developers. Georges Saab is an engineer with Oracle who has been working on Java for more than a decade. He joins the show to discuss how Java development patterns are changing, and how the language is evolving to accommodate those changes, including discussion of garbage collection and dynamic application class data sharing.
#1291
Dec 2, 2019 · 1 hr 1 min
Distributed SQL with Karthik Ranganathan and Sidharth Choudhury
Relational databases provide durable transactional systems for storing data. The relational model has existed for decades, but the requirements for a relational database have changed. Modern applications have requirements for high volumes of data that do not fit onto a single machine. When a database gets too big to fit on a single machine, that database needs to be sharded into smaller subsets of the data. These database shards are spread across multiple machines, and as the database grows, the database can be resharded to scale to even more machines. To ensure durability, a database needs to be replicated. The database needs to be able to survive any single machine losing power or getting destroyed. Sharding and replication allow a relational database to be scalable, durable, and highly available. There are many ways to build sharding and replication into a database. Karthik Ranganathan and Sidharth Choudhury are engineers with YugabyteDB, a distributed SQL database. In today’s episode, we discuss the modern requirements of a distributed SQL database, and compare the applications of distributed SQL to those of other systems such as Cassandra and Hadoop. We also talk through the competitive market of cloud-based distributed SQL providers such as Google Cloud Spanner and Amazon Aurora. YugabyteDB is an open source database that competes with these other relational databases. Full disclosure: YugabyteDB is a sponsor of Software Engineering Daily.
#1290
Nov 29, 2019 · 1 hr 3 min
Economics of Software with Russ Roberts Holiday Repeat
Originally published July 14, 2016 EconTalk is a weekly economics podcast that has been going for a decade. On EconTalk, Russ Roberts brings on writers, intellectuals, and entrepreneurs for engaging conversations about the world as seen through the lens of economics. Russ Roberts is today’s guest, and it is a treat because I have been listening to EconTalk since 2006 and it was a central point of inspiration for what Software Engineering Daily has become. On this episode, we talk about how software impacts the world economically, from bitcoin’s promise of zero cost transactions to the opportunities and regulatory challenges of the software-enabled gig economy.
#1289
Nov 28, 2019 · 58 min
Uber’s Data Platform with Zhenxiao Luo Holiday Repeat
Originally published May 24, 2018 When a user takes a ride on Uber, the app on the user’s phone is communicating with Uber’s backend infrastructure, which is writing to a database that maintains the state of that user’s activity. This database is known as a transactional database or “OLTP” (online transaction processing). Every active user and driver and UberEATS restaurant is writing data to the transactional data store. Periodically, that data is copied from the transactional data system to a different data storage system, where that data can be queried for large-scale data analysis. For example, if a data scientist at Uber wants to get the average amount of miles that a given user rode in February, that data scientist would issue a query to the analytical data cluster. Uber uses the Hadoop distributed file system (HDFS) to store analytical data. On this file system, Uber has a version history of all of the company’s useful historical data. Trip history, rider activity, driver activity–every data point that is in the transactional database–but in a file format that is easier to query for large scale processing. This file format is known as Parquet. Data scientists, machine learning engineers, and real-time application developers all depend on the massive quantities of data that are stored in these Parquet files on Uber’s HDFS cluster. To simplify the access of that data by many different clients, Uber uses Presto, an analytical query engine originally built at Facebook. Presto translates SQL queries into whatever query language is necessary to access the underlying storage medium–whether that storage system is an ElasticSearch cluster, a set of Parquet files, or a relational database. Presto is useful because it simplifies the relationship between data engineers and the application developers who are building on top of the data engineering infrastructure. In today’s show, Zhenxiao Luo joins to give an end-to-end description of Uber’s data infrastructure–from the ingest point of the OLTP database to the OLAP data storage system on HDFS, to the wide range of data systems and applications that run on top of that OLAP data.
#1288
Nov 27, 2019 · 59 min
Kong API Platform with Marco Palladino Holiday Repeat
Originally published January 4, 2019 When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging. API gateways have grown in popularity as applications have become more distributed, and companies offer a wider variety of services. If an API is public, and anyone can access it, you might need to apply rate limiting so that users cannot spam the API. If the API is private, the user needs to be authenticated before the request is fulfilled. Kong is a company that builds infrastructure for API management. The Kong API gateway is a widely used open source project, and Kong is a company built around supporting and building on top of the API gateway. Marco Palladino is the co-founder and CTO of Kong. He joins the show to tell the story of starting Kong eight years ago, and how the API gateway product evolved out of an API marketplace. Marco also discusses the architecture of Kong and his vision for how the product will develop in the future–including the Kong service mesh.
#1287
Nov 26, 2019 · 58 min
Future of Computing with John Hennessy Holiday Repeat
Originally published June 7, 2018 Moore’s Law states that the number of transistors in a dense integrated circuit doubles about every two years. Moore’s Law is less like a “law” and more like an observation or a prediction. Moore’s Law is ending. We can no longer fit an increasing amount of transistors in the same amount of space with a highly predictable rate. Dennard scaling is also coming to an end. Dennard scaling is the observation that as transistors get smaller, the power density stays constant. These changes in hardware trends have downstream effects for software engineers. Most importantly–power consumption becomes much more important. As a software engineer, how does power consumption affect you? It means that inefficient software will either run more slowly or cost more money relative to our expectations in the past. Whereas software engineers writing code 15 years ago could comfortably project that their code would get significantly cheaper to run over time due to hardware advances, the story is more complicated today. Why is Moore’s Law ending? And what kinds of predictable advances in technology can we still expect? John Hennessy is the chairman of Alphabet. In 2017, he won a Turing award (along with David Patterson) for his work on the RISC (Reduced Instruction Set Compiler) architecture. From 2000 to 2016, he was the president of Stanford University. John joins the show to explore the future of computing. While we may not have the predictable benefits of Moore’s Law and Dennard scaling, we now have machine learning. It is hard to plot the advances of machine learning on any one chart (as we explored in a recent episode with OpenAI). But we can say empirically that machine learning is working quite well in production. If machine learning offers us such strong advances in computing, how can we change our hardware design process to make machine learning more efficient? As machine learning training workloads eat up more resources in a data center, engineers are developing domain specific chips which are optimized for those machine learning workloads. The Tensor Processing Unit (TPU) from Google is one such example. John mentioned that chips could become even more specialized within the domain of machine learning. You could imagine a chip that is specifically designed for a LSTM machine learning model. There are other domains where we could see specialized chips–drones, self-driving cars, wearable computers. In this episode, John describes his perspective on the future of computing, and offers some framework for how engineers can adapt to that future.
#1286
Nov 25, 2019 · 1 hr 6 min
Indie Hackers with Courtland Allen Holiday Repeat
Originally published November 4, 2016 Indie Hackers is a website that profiles independent developers who have made profitable software projects, usually without raising any money. These projects make anywhere from a few hundred dollars a month to more than $100,000 as in the case with park.io, one of the services profiled by Indie Hackers. Courtland Allen is the creator, engineer, and interviewer behind Indie Hackers. For each business that is profiled by Indie Hackers, Courtland conducts a short interview with the founder. Courtland joins the show to discuss the changing trends that are making it easier to bootstrap a software business if you are a capable developer–or even if you are a nontechnical person who understands how software works. Since Courtland and I are both in the business of interviewing engineers, we had a lot to talk about, and this is a fantastic episode.
#1285
Nov 23, 2019 · 50 min
Cloud Dependencies with Mya Pitzeruse
New software abstractions always take advantage of the abstractions that have been built before. Software libraries allow us to import code that sits on the same host as a new program. Open source software let us copy and paste existing code, or clone entire repositories. Cloud providers offer hosted tools and APIs that we can leverage to develop scalable, easy-to-use infrastructure. When existing pieces of software are built into new software, the existing software becomes a dependency. Managing those dependencies is an engineering problem. Mya Pitzeruse is the founder of deps.cloud, a project with the goal of improving dependency changes across a company’s ecosystem. In today’s show, we talk about the modern dependency issues of a large company, and her perspective on how to address them. Mya has developed the project in public on FindCollabs, and we also spend some time talking about building in the open.
#1284
Nov 22, 2019 · 46 min
Web Application Testing with Gabriel-James Safar
Web applications are used on a wide variety of platforms. On each of these platforms the web app needs to load properly and allow the user to navigate the website and interact with all of the user flows, such as sign-up, login, and the various read and write operations that make up the functionality of any website. It is difficult to ensure web application functionality across all platforms, because there are so many platforms. Different operating systems, different underlying hardware, different browsers, and different device form factors all create potential sources of suboptimal website functionality and performance. Testing web applications often involves the work of a manual quality assurance (QA) employee. The QA can simulate the procedures that a normal user would go through. This QA process ensures that the website is operating as expected. But the manual workflow can slow down software development. Gabriel-James Safar is a software engineer and the founder of Madumbo, which was acquired by Datadog. Madumbo was founded with the goal of making web application testing simpler by identifying errors in pages and enabling users to create test suites from recordings of user activity. This process simplifies and accelerates the testing process. Gabriel-James joins the show to talk about his experience building Madumbo, and his perspective on the modern application testing process. Full disclosure: Datadog is a sponsor of Software Engineering Daily.
#1283
Nov 21, 2019 · 1 hr 2 min
HTTP with Julia Evans
HTTP is a protocol that allows browsers and web applications to communicate across the Internet. Everyone knows that HTTP is doing some important work, because “HTTP” is at the beginning of most URLs that you enter into your browser. You might be familiar with the request/response model, and HTTP request methods such as GET, PUT, and POST. But unless you have had a reason to learn more about the details of HTTP, you probably don’t know much more than that. Julia Evans is a software engineer and writer who creates Wizard Zines, a series of easy-to-read online magazines that explain technical software topics. Julia’s zines include “Linux Debugging Tools”, “Help! I Have A Manager!”, and recently “HTTP: Learn your browser’s language”. Her zines are a creative, innovative format for describing the world of software engineering while also exploring her own artistic pursuits in writing, design, and illustration. Julia was previously on the show to discuss Ruby profiling, and she returns to the show to discuss HTTP, as well as her creative process and goals with Wizard Zines.
#1282
Nov 20, 2019 · 1 hr 1 min
Indie Hack or Venture Back with Lynne Tye
Key Values is a platform where companies are profiled with descriptions of their company values. These profiles describe features such as work-life balance, company culture, daily routines, and strategy. Lynne Tye created Key Values with the goal of building a small business that would make money through connecting job seekers to companies with a culture that matched their own personal values system. Key Values has become highly successful, and Lynne is making enough money from the business to live comfortably. In a previous episode, Lynne and I discussed her founding story and the engineering of Key Values. Today’s episode picks up a few years later, with Lynne having found significant success with her own company. Lynne’s software business is an example of a growing trend: “Indie Hackers”. This trend was identified by Courtland Allen, founder of the Indie Hackers platform. Courtland is close friends with Lynne, and Lynne’s desire to start her own software company was influenced by her conversations with Courtland. At a certain point, Lynne was considering raising money and growing Key Values. She was accepted into Y-Combinator. But she decided to stick with the Indie Hackers route, and grow the business independently. Lynne joins the show to talk about the process of starting a software business, and the pivotal decisions she has made around financing, growth, and her own psychology.
#1281
Nov 19, 2019 · 1 hr 7 min
Bubble: Code-Free Programming with Emmanuel Straschnov and Joshua Haas
The vision of code-free programming has existed for decades. Software engineers have always dreamed of empowering non-technical users with the same creative tools that programmers have access to. For many years, the underlying technology of the web was not powerful enough to make this dream a reality. Platforms such as WordPress, Squarespace, and Wix have allowed for some of the functionality of programming without writing code, but the scope of what those tools could accomplish was limited. Today, the web has caught up. Improvements in browser technology and client devices mean that the end user has access to more powerful technology. The API economy encapsulates large amounts of functionality into cheap, well-defined functionality. No-code platforms are finding success among developers and non-developers, after persisting for several years as their technology matured. Bubble is a code-free platform for building websites, startups, and internal tools. Emmanuel Straschnov and Joshua Haas are the founders of Bubble, and they join the show to tell the story of Bubble, and give their perspective on engineering, business, and the low-code movement. The story of Bubble is strikingly similar to that of no-code tools Airtable and Webflow, which we have covered in previous episodes. All of these products have taken years to get to maturity, with no shortcuts–only gritty, difficult engineering problems and performance improvements. Each of these no-code platforms has an inspiring story behind them, and persistent founders who eventually got their product to success.
#1280
Nov 18, 2019 · 55 min
Cloudflare Serverless with Zack Bloom
“Serverless” is an execution model where applications are scheduled and deployed to servers that are not directly managed by the application developer. In serverless execution, an application only loads and operates when a user actually needs to get a response from that application. This saves on resources, because many applications do not need to run at all times–they only need to be available for user requests. The serverless model was popularized by Amazon Web Services Lambda. When Lambda first launched in 2015, it was an experimental product. Today, it is a widely used product and the market has validated the desire for serverless execution. Other cloud providers have introduced different models of serverless functionality including Google Cloud Functions, Azure Functions, and Fastly edge computing. Zack Bloom is director of product for product strategy at Cloudflare, and he joins the show to discuss Cloudflare’s model for serverless execution. Zack also discusses Cloudflare’s growing product line, including the fast, privacy-protecting DNS resolver 1.1.1.1. Zack is a rare mix of engineering, business strategy, and product vision, which made for a great conversation. For more content, you can check out our episodes about serverless technology and episodes about Cloudflare.
#1279
Nov 15, 2019 · 1 hr 13 min
Dagster with Nick Schrock
Data engineering is difficult. Companies want to be able to maximize the value they get from their large data sets, but there are so many steps required for deriving that value that most companies feel like they are always far behind the ideal state of where they could be. The cloud makes it cheap to save data. Tools like Spark and Snowflake give us usable APIs for simplifying the data processing. Workflow engines like Airflow help us visualize a complex execution path of a data pipeline. With all of this tooling, why is it so hard to make use of our data? Nick Schrock is the creator of Dagster, an open source system for building modern data applications. Nick is also the CEO of Elementl, a company that he is building around Dagster. Before creating Dagster, Nick worked at Facebook, where he co-created GraphQL. Nick returns to the show to discuss modern data engineering, and why it continues to be so difficult for engineers to be productive with their data.

Showing 1,061–1,080 of 2,221 episodes