NewsTech News

Software Engineering Daily

softwareengineeringdaily.com

Software Daily: The world through the lens of software.

Play

Open in

2,221 episodes
Avg 53 min
English

Episodes 2,221

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

#1278
Nov 14, 2019 · 58 min
GraalVM Quarkus: Java Acceleration with Guillaume Smet and Emmanuel Bernard
Java programs run in a different environment than they did ten years ago. Modern infrastructure runs on containers sitting in a Kubernetes cluster. The optimal configuration for a Java program in that context is different than it was for an environment dominated by virtual machines and bare metal. When you are co-scheduling your services with each other, those services could be fighting for resources. You may want to optimize them with more ahead-of-time compilation. Quarkus is a system for accelerating Java performance through the use of GraalVM. In a previous show, we explored the basics of GraalVM. In today’s show, Guillaume Smet and Emmanuel Bernard join the show to describe an application of GraalVM: the acceleration of Java. Guillaume and Emmaneul are engineers at Red Hat, and are working on changes to the Java ecosystem that are informed by the cloud and the rise of Kubernetes.
#1277
Nov 13, 2019 · 58 min
Bugsnag Business with James Smith
Crash monitoring emerged as a software category over the last decade. Crash monitoring software allows developers to understand when their applications are crashing on client devices. For example, we have an app for Software Engineering Daily that people download on Android or iOS. Users download the app to their smartphone. When the user is playing an episode, and the app crashes, the details of the crash are sent to a server that collects all of these crash reports. Crash reports allow a company to understand where their application is breaking on client devices. This is important, since there are so many client surfaces to test, from iOS to Android to browsers. As a business, crash monitoring is a category that has some similarities to log management. There are lots of companies that offer crash monitoring. At first glance, it seems like a simple problem to solve. It seems like a market without winner-take-all or winner-take-most dynamics. But at scale, crash monitoring becomes a deeply complex engineering problem. From indexing to database choices to complex distributed systems tradeoffs, crash monitoring is not a simple business, and it promises to provide an extremely good business for the few companies who are able to out-execute the crowded market. James Smith is the CEO of Bugsnag, a company that makes crash monitoring and application stability tools. James returns to the show to discuss the growth-stage engineering challenges of error monitoring, and the business opportunities that come with them.
#1276
Nov 12, 2019 · 48 min
Incident Response Machine Learning with Chris Riley
Software bugs cause unexpected problems at every company. Some problems are small. A website goes down in the middle of the night, and the outage triggers a phone call to an engineer who has to wake up and fix the problem. Other problems can be significantly larger. When a major problem occurs, it can cause millions of dollars in losses and requires hours of work to fix. When software unexpectedly breaks, it is called an incident. To triage these incidents, an engineer uses a combination of tools, including Slack, GitHub, cloud providers, and continuous deployment systems. These different tools emit updates that can be received by an incident response platform, which allow the on-call engineer to have the information they need centralized to more easily work through the incident. On-call rotation means that different people will be responsible for dealing with different incidents that occur. When an incident happens, the current engineer who is on-call may not be aware that a similar incident happened last week. It might be easier for the new engineer to triage the issue if they have insights about how the incident was managed during the first time. Chris Riley is a DevOps advocate with Splunk. He joins the show to discuss the application of machine learning to incident response. We discuss the different data points that are created during an incident, and how that data can be used to build models for different types of incidents, which can generate information to help the engineer respond appropriately to an incident. Full disclosure: Splunk is a sponsor of Software Engineering Daily.
#1275
Nov 11, 2019 · 1 hr
Gatsby with Kyle Mathews and Sam Bhagwat
Frontend software development has become as complex as backend development. There was a time when frontend web development was simple. There was a small number of JavaScript frameworks and templating systems. Your CSS was simple configuration for the colors on your webpage. Today, there is a giant ecosystem of frontend tools, APIs, and middleware delivering data to the user. Gatsby is a framework based on React that allows developers to build performant web applications. Gatsby is not easy to explain. In some ways, it is like a compiler for your website. Gatsby pulls in the data that you need to build your website, including CMS data, APIs, and markdown, and then links that information into React components and CSS. This happens on the server, so your user gets served a website that does not require lots of round trips as your website renders. Kyle Mathews and Sam Bhagwat are the founders of Gatsby, the company that is based around GatsbyJS. They join the show to describe their vision for the framework, and their vision for the company.
#1274
Nov 8, 2019 · 44 min
Drones with Chris Anderson
Drones will deliver food to us. Drones will be able to extinguish fires. Drones will be used to relay Internet signals and make the world more connected. These all sound like great ideas, so why aren’t there more drones in the sky today? There are many answers to that question, some of which relate to engineering and some of which are about the regulatory barriers. Chris Anderson is the CEO of 3D Robotics, a drone company which he started seven years ago. Before 3DR, Chris worked for many years as a journalist writing about technology and science. He was the editor-in-chief at Wired for 11 years, a writer for The Economist for 7 years, and spent 3 years at both of the leading scientific journals, Nature and Science. Chris also wrote The Long Tail, the influential 2004 book which described a set of emergent Internet trends. Chris joins the show for a discussion about drones, journalism, and his perspective on modern technology.
#1273
Nov 7, 2019 · 54 min
LinkedIn Data Engineering with Kapil Surlaker
A large social network needs to develop systems for ingesting, storing, and processing large volumes of data. Data engineering at scale requires multiple engineering teams that are responsible for different areas of the infrastructure. Data needs to be structured coherently in order to minimize the data cleaning process. Machine learning models need to be developed, deployed, and iterated on at scale. Areas of the company which produce data need to be decoupled from the areas of the company which consume data, so that engineers throughout the company can reliably build tools on top of these large data sets. In our previous episodes about LinkedIn, we covered two major components of LinkedIn’s data engineering systems: the Kafka infrastructure and the LinkedIn data platform used by engineers to productively build data applications. Kapil Surlaker is a senior director of engineering at LinkedIn, and he joins the show to discuss the bigger picture of LinkedIn’s data infrastructure. Kapil works with teams across LinkedIn to understand the requirements for the products and internal tools, and translate those requirements into team structures and software platforms that let LinkedIn use data more productively. We discuss a wide range of topics, including engineering management, the modern data platform, and LinkedIn’s adoption of public cloud. Full disclosure: LinkedIn is a sponsor of Software Engineering Daily.
#1272
Nov 6, 2019 · 57 min
FOSSA: Open Source Management with Kevin Wang
Every company has a software supply chain. A company builds its products from custom code, paid APIs, paid proprietary binaries, and open source software libraries. As the types of software available have increased, the management of the software supply chain has become complex. Large software companies have always needed to ensure the security of their software. With the growing variety of open source licenses, these companies also have to deal with an increased set of legal complexities. If an open source project is used in a way that violates an open source license, the company is subject to legal risk. FOSSA is a company that focuses on automating the management of open source compliance and security. Kevin Wang is the CEO of FOSSA, and he joins the show to discuss the modern issues of software licensing and his experience building a company.
#1271
Nov 5, 2019 · 1 hr 3 min
Scale with Alexandr Wang
Machine learning is widely understood by the software community. But it is still hard to build a company around machine learning, because there is not easy access to large, unique data sets. Scale is a platform for training and validating data that is used for machine learning. Most machine learning models are built with supervised learning. Labeled examples are analyzed to understand the mathematical correlations between those labels. The more labeled training examples there are, the more accurate the correlations will be. Today, we have high quality frameworks for writing the models. We have cheap cloud computing for training and deploying the models. The biggest factor that is preventing a wide variety of potential machine learning applications from existing is lack of access to large, labeled data sets. Scale gives developers an API for labeling images, sound, natural language, and video. Scale is used by self-driving car companies, Airbnb, OpenAI, retailers, and robotics companies. The product is used broadly and at high volume. Scale was started only three years ago, and recently raised $100m at a valuation above $1b, making it one of the fastest growing software companies in history. Alexandr Wang joins the show to discuss how Scale works, the future of machine learning, and the future of work. He also describes the complexities of building Scale, and how he manages his own psychological state.
#1270
Nov 4, 2019 · 50 min
Leadership with Ben Horowitz
Photo credit: Elisabeth Fall Ben Horowitz started Loudcloud with Marc Andreessen in 1999. He ran the company for eight years and chronicled his experience in his first book The Hard Thing About Hard Things. In his time running Loudcloud, the dotcom bubble burst, but Loudcloud needed cash so badly that he took the company public in 2001. Loudcloud went through layoffs, downsizing, and a difficult strategic maneuver in which Loudcloud sold its cloud provider business for cash, then used the core competency it had developed to create new software for building and running cloud services. This new software was the core product of the company Opsware, which was sold to HP in 2007 for $1.6bn. The Loudcloud story looks like a rational, straightforward execution in retrospect, but at many points in the timeline, Ben was unsure he was making the correct decision. As the subtitle of his first book states “there are no easy answers.” The Hard Thing About Hard Things tells the story of Loudcloud and Opsware in harrowing detail. Most founders of software companies will end up reading the book at some point when they are building their company, because there are so few books which capture the granular details of what it feels like to run a company. A CEO is completely alone in their understanding of the company. Nobody else has nearly as much information as the CEO–not the board, not the market, and not the employees. When you are a CEO, there is simply nobody to turn to who can give you the actionable advice that you wish you could have access to. And because there is nobody else, it means that the CEO’s own psychological state is extremely important. The Hard Thing About Hard Things provides a CEO with solace: while the CEO is alone within their company, they are not alone in the world. Every CEO has a set of issues which they have never faced before, and the CEO can learn to face those issues confidently and competently. Like any influential book, The Hard Thing About Hard Things presents the reader with useful answers, but also raises many questions. How can a normal person foster the mentality of a leader? How can a leader convince smart people to follow their direction? How can a seemingly crazy direction be framed as completely rational? The second book by Ben Horowitz is called What You Do Is Who You Are. This book surveys a set of case studies in leadership, including a Haitian slave revolt, the Mongol empire, and a dominant prison gang. By studying violent environments, Ben frames leadership in the context of the highest stakes. These stories are about life and death. When a leader’s performance is measured in blood, it frames the true nature of leadership in the starkest resolution. Ben uses each distilled example as a base case which inducts into broader applications: the cultures of Netflix, Facebook, Uber, and McDonalds are explored alongside editorials about Hillary Clinton and hip hop culture. Throughout all of these stories, the most important thread is continually reinforced: the leader creates the culture. The culture is the leader. What you do is who you are. Ben joins the show to discuss his writing, and how he has applied these beliefs to Andreessen Horowitz, the venture capital firm he co-founded and leads today.
#1269
Nov 1, 2019 · 1 hr 9 min
Facebook Leadership with Arturo Bejar
Facebook leadership has a significant amount of engineers in its ranks, and engineers understand how to create an environment that appeals to other engineers. Engineers do not like working on projects that they are not interested in, so Facebook optimizes for matching engineers to enjoyable work. Engineers do not like taking orders from managers, so Facebook optimizes for a collaborative relationship between engineering and management. Arturo Bejar was a director of engineering at Facebook for more than five years. Arturo managed a team that built site integrity tools, as well as the Product Infrastructure team, which created the groundbreaking React and GraphQL open source tools. Arturo has a deep understanding of what engineers want from a manager, and this is apparent in the results of the teams he has worked with. Arturo also cares deeply about the human side of technology. When he was at Facebook, he started the Compassion Project, which looked at ways that technology could help support people through difficult times, and alleviate suicide and bullying. Arturo joins the show for a conversation about engineering management, humanity, and the ethos of Facebook. Articles by Arturo Bejar Managing Your Manager Challenges and Opportunities in Communication Online: Summary and Full Version
#1268
Oct 31, 2019 · 1 hr 7 min
Facebook Fallout with Antonio Garcia Martinez
Chaos Monkeys is a book about Silicon Valley startups and Facebook. It is one of the most accurate books written about the modern technology industry, and captures both the negatives and the positives of software companies. Antonio Garcia Martinez is the author of Chaos Monkeys. He wrote the book after going through a gauntlet of prototypical Silicon Valley experiences. Antonio founded an adtech company called AdGrok. His company was funded by the startup accelerator Y-Combinator, and during his time at the company there were many dramatic events that Antonio tells in great detail. AdGrok was acquired by Twitter, and Antonio went to work at Facebook on its ads platform. Antonio is a fantastic writer. What makes Chaos Monkeys special is that it reads like a book written by an author who accidentally found himself in a technology career, rather than a technologist who opportunistically wrote a book. Note: This episode contains explicit language.
#1267
Oct 30, 2019 · 1 hr 7 min
Facebook Reflections with Pete Hunt and Nick Schrock
Facebook engineering is unique. Software is built at Facebook in a way that is distinctly different than any other company. In our series of shows about Facebook engineering, we have mostly covered the positive side of Facebook’s practices. In today’s show, we explore the downsides. Facebook moves fast. Engineers within the company must move fast, which can reduce the time spent on formal process and documentation. A fast pace can lead to a chaotic environment. In this kind of environment, not every employee thrives, and product development can sometimes suffer. But there are clear benefits to moving fast, and many software organizations could benefit from moving faster. Pete Hunt was an engineer with Facebook who worked on Facebook Video, Instagram, and React. Nick Schrock worked on core infrastructure at Facebook and was the co-creator of GraphQL. Pete and Nick helped create the Facebook culture, and are aware of the best and worst aspects of it. They join the show to reflect on their time at Facebook, and the downsides of moving fast.
#1266
Oct 29, 2019 · 53 min
Facebook Products with Peter Deng
Peter Deng has worked on most of Facebook’s major products: Newsfeed, Instagram, Oculus, and Messenger. These different products have different requirements, but are all part of the same ethos of connecting people through social networks. Facebook is a consumer product company that is powered by a strong engineering workforce. The relationship between product managers and engineering are two parts of a three-legged relationship: product, engineering, and design. Every major product within Facebook is built with the teamwork of product, engineering, and design. After almost ten years at Facebook, Peter joined Uber as the Head of Rider. At Uber, Peter works on a very different platform: a real-world, two-sided marketplace. Every change to the Uber platform has an impact on the economic relationship between riders and drivers. This creates a set of product development constraints that contrast with the social network of Facebook. Peter joins the show to share how he thinks about product management, and how the core competencies of a business inform product strategy.
#1265
Oct 28, 2019 · 1 hr 7 min
Facebook Entrepreneurship with Jeff Rothschild
Jeff Rothschild was one of the earliest engineers to join Facebook. In the 1990s, Jeff had co-founded Veritas Software and helped it to its IPO in 2004. After Veritas, Jeff worked on several other of his own companies. He was working with Accel Partners on investments when he started to learn about Facebook. Accel was an investor in Facebook, having made a large investment in 2005. After the investment, Jeff was helping the company with hiring. As Facebook built out its senior engineering leadership, Jeff realized that he would enjoy working with the company in a more active role. For the next ten years, Jeff Rothschild worked as a VP of infrastructure software at Facebook. He helped scale the company’s infrastructure and architect the technical strategy. By 2005, Jeff’s entrepreneurial background included software companies ranging from storage management to gaming. His diverse skill set was useful for setting product direction as well as solving novel engineering problems. Jeff joins the show to discuss the early days of Facebook and his philosophy of entrepreneurship.
#1264
Oct 25, 2019 · 45 min
Alluxio: Data Orchestration with Haoyuan Li
In 2009, the Berkeley AMPLab was a center of innovation. Three projects from AMPLab have turned into successful open source projects and companies: Spark, Mesos, and Alluxio. Haoyuan Li was the creator of Alluxio, and he returns to the show to discuss his journey taking Alluxio from a research project to a company that has customers including Alibaba, Baidu, Wells Fargo, and Samsung. Alluxio is a distributed, fault-tolerant, tiered storage system. Alluxio allows application developers to think in terms of the latency that they require from their infrastructure rather than the details of different storage systems. Haoyuan discusses the process of integrating with gigantic companies like cloud providers, telecoms, and huge ecommerce companies. Alluxio is also hosting an upcoming conference, the Data Orchestration Summit November 7th at the Computer History Museum in Mountain View California. November 7, 2019 is the first open source Data Orchestration Summit in the Computer History Museum in Mountain View, CA. The summit brings together data engineers, data platform engineers, and data scientists to share their challenges and learnings from building and using modern analytics, AI, and cloud technologies. Featuring tech talks covering use cases, demos, best practices, and tutorials by industry experts from EA, Walmart, DBS Bank, Netflix, AWS, Rakuten, Tencent, Google, Baidu, Alibaba and more, with a focus on how to build cloud native analytics & AI platforms. Use code “SDE” to get a 50% discount.
#1263
Oct 24, 2019 · 56 min
Redis with Alvin Richards
Redis is an in-memory database that persists to disk. Redis is commonly used as an object cache for web applications. Applications are composed of caches and databases. A cache typically stores the data in memory, and a database typically stores the data on disk. Memory has significantly faster access times, but is more expensive and is volatile, meaning that if the computer that is holding that piece of data in memory goes offline, the data will be lost. When a user makes a request to load their personal information, the server will try to load that data from a cache. If the cache does not contain the user’s information, the server will go to the database to find that information. Alvin Richards is chief product officer with Redis Labs, and he joins the show to discuss how Redis works. We explore different design patterns for making Redis high availability, or using it as a volatile cache, and we talk through the read and write path for Redis data. Full disclosure: Redis Labs is a sponsor of Software Engineering Daily.
#1262
Oct 23, 2019 · 55 min
LinkedIn Data Platform with Carl Steinbach
LinkedIn is a social network with petabytes of data. In order to store that data, LinkedIn distributes and replicates that data across a large cluster of machines running the Hadoop Distributed File System. In order to run calculations across its large data set, LinkedIn needs to split the computation up using MapReduce-style jobs. LinkedIn has been developing its data infrastructure since the early days of the Hadoop ecosystem. LinkedIn started using Hadoop in 2008, and in the last 11 years, the company has adopted streaming frameworks, distributed databases, and newer execution runtimes like Apache Spark. With the popularization of machine learning, there are more applications for data engineering than ever before. But the tooling around data engineering means that it is still hard for developers to find data sets, clean their data, and build reliable models. Carl Steinbach is an engineer at LinkedIn working on tools for data engineering. In today’s episode, Carl discusses the data platform inside LinkedIn, and the strategies that the company has developed around storing and computing large amounts of data. Full disclosure: LinkedIn is a sponsor of Software Engineering Daily.
#1261
Oct 22, 2019 · 1 hr 2 min
Crypto Businesses with Haseeb Qureshi
Cryptocurrencies are a fundamental computer science invention. Cryptocurrencies crashed in 2018 but the technology remains as promising as ever. Bitcoin is a decentralized currency, and a plausible end state that is implied by Bitcoin’s current trajectory is a permissionless, decentralized financial system. This idea of decentralized finance or “DeFi” begs numerous questions: who will build the companies that provide the infrastructure for decentralized finance? Who will be the lenders? Who will be the credit agencies? Who will be the escrow services? How big will the teams need to be? Will these systems be built on smart contracts, or can it be done with centralized cloud providers? Haseeb Qureshi is a managing partner with DragonFly Capital and a frequent guest on SE Daily. He returns to the show to discuss his thesis on what kinds of crypto companies make sense, and how he thinks about investments into crypto startups. Haseeb recently wrote an article about how to build a crypto startup.
#1260
Oct 21, 2019 · 1 hr 3 min
Dark Lang with Ellen Chisa and Paul Biggar
Dark Lang is a programming language that is tightly integrated with the cloud. Dark takes an opinionated approach that most developers are going to want to run their applications in the cloud, and this perspective influences how Dark looks at deployments, IDEs, exception handling, and other aspects of software development. Paul Biggar is the founder of CircleCI, and ran the company for eight years before leaving to found Dark with Ellen Chisa. Ellen is a software engineer and the CEO of Dark. Paul and Ellen join the show to give their perspective on modern software engineering, and why it was time to build a new high level language that assumes the presence of a cloud. It is difficult to get programmers to adopt a new language. It is even harder to get those programmers to pay for products built around that language. But the timing could be perfect for Dark. Software development is undergoing tremendous change–and many of these changes work to Dark’s advantage, such as the growing adoption of feature flags, low code tools, and sophisticated continuous delivery workflows. Whether or not Dark is a success, it is a bold project, and the team is working on something they believe in. We also discussed AWS–and whether the largest cloud provider has an obligation to contribute back to the open source community.
#1259
Oct 18, 2019 · 59 min
LinkedIn Kafka with Nacho Solis
Apache Kafka was created at LinkedIn. Kafka was open sourced in 2011, when the company was eight years old. By that time, LinkedIn had developed a social network with millions of users. LinkedIn’s engineering team was building a range of externally facing products and internal tools, and many of these tools required a high-throughput system for publishing data and subscribing to topics. Kafka was born out of this need. Over time, Kafka’s importance within LinkedIn has only grown. Kafka plays a central role for services, log management, data engineering, and compliance. LinkedIn might be the biggest user of Apache Kafka in the entire software industry. Kafka has many use cases, and it is likely that they are almost all on display within LinkedIn. Nacho Solis is a senior software engineering manager at LinkedIn, where he helps teams build infrastructure for Kafka, as well as Kafka itself. Nacho joins the show to discuss the history of Kafka at LinkedIn, and the challenges of managing such a large deployment of Kafka. We also talk about streaming, data infrastructure, and more general problems in the world of engineering management. Full disclosure: LinkedIn is a sponsor of Software Engineering Daily.

Showing 1,081–1,100 of 2,221 episodes