NewsTech News

Data Science Tech Brief By HackerNoon

HackerNoon

Learn the latest data science updates in the tech world.

Play

Open in

36 episodes
Avg 11 min
English

Episodes 36

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

Yesterday · 28 min
Building an Auditable Pipeline for Cross-Language News Coverage Analysis
This story was originally published on HackerNoon at: https://hackernoon.com/building-an-auditable-pipeline-for-cross-language-news-coverage-analysis. How I turned 2,565 Arabic news reports on Lebanon’s war into 234 auditable incidents with Python, Media Cloud, and a traceable data pipeline. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #data-pipeline, #media-analysis, #data-provenance, #cross-language-information, #python-data-pipelines, #temporal-data-normalization, #data-reconciliation, and more. This story was written by: @soufan. Learn more about this writer by checking @soufan's about page, and for more stories, please visit hackernoon.com. The project aimed to connect international news coverage to specific fatal strikes in a small Lebanese town by reconstructing what had happened accurately enough to search for it in international news databases. The pipeline was built to turn a pile of Arabic news bulletins into a clean, traceable chain of incidents, locations, and dates.
Yesterday · 8 min
How Shallow and Deep Copies Work in Python
This story was originally published on HackerNoon at: https://hackernoon.com/how-shallow-and-deep-copies-work-in-python. I copied a Python dictionary, but the original still changed. Here's how shallow copies, shared references, and deep copies actually work. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #python-programming, #python-dictionaries, #shallow-copy, #deep-copy, #python-mutability, #object-references, #python-debugging, #mutable-default-arguments, and more. This story was written by: @sohelalam79. Learn more about this writer by checking @sohelalam79's about page, and for more stories, please visit hackernoon.com. `dict.copy()` only copies the outer dictionary nested lists/dicts stay shared, so mutating them changes both copies. It's not a bug, it's shallow copy working as designed. Use `deepcopy()` only when you need full independence.
Monday · 9 min
How I Fixed a Data Ingestion Bottleneck With Date Partitioning
This story was originally published on HackerNoon at: https://hackernoon.com/how-i-fixed-a-data-ingestion-bottleneck-with-date-partitioning. I fixed a recurring data ingestion bottleneck by working with a vendor to introduce date-partitioned exports and reduce unnecessary file discovery. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #distributed-systems, #data-ingestion-pipelines, #data-partitioning, #object-storage, #incremental-data-ingestion, #idempotent-data-processing, #data-pipeline-reliability, and more. This story was written by: @rahuln. Learn more about this writer by checking @rahuln's about page, and for more stories, please visit hackernoon.com. Our vendor-data ingestion jobs were repeatedly enumerating a growing history of export files. I worked with the vendor's engineers on date-partitioned delivery and prepared our consumer in parallel. This case study explains how partitioning bounds file discovery, why event-date partitions need late-arrival and replay policies, and how to coordinate the producer-consumer cutover.
October 1 · 55 min
224 Blog Posts To Learn About Web Scraping
This story was originally published on HackerNoon at: https://hackernoon.com/224-blog-posts-to-learn-about-web-scraping. Learn everything you need to know about Web Scraping via these 224 free HackerNoon blog posts. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #web-scraping, #learn, #learn-web-scraping, and more. This story was written by: @learn. Learn more about this writer by checking @learn's about page, and for more stories, please visit hackernoon.com.
September 17 · 4 min
Mastering Databricks: A Developer’s Guide to Lakehouse Architecture & PySpark Pipelines
This story was originally published on HackerNoon at: https://hackernoon.com/mastering-databricks-a-developers-guide-to-lakehouse-architecture-and-pyspark-pipelines. A complete developer guide to Databricks and Lakehouse architecture. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #databricks, #data-engineering, #apache-spark, #pyspark, #machine-learning, #architecture, #lakehouse-architecture, #pyspark-pipelines, and more. This story was written by: @jagan_489. Learn more about this writer by checking @jagan_489's about page, and for more stories, please visit hackernoon.com. Whether you're building real-time streaming pipelines or deploying production generative AI applications, this guide breaks down the core architecture, implementation patterns, and code needed to master Databricks.
September 11 · 11 min
1.89 Seasons: A Baseball Experiment About Hiring and Human Judgment
This story was originally published on HackerNoon at: https://hackernoon.com/189-seasons-a-baseball-experiment-about-hiring-and-human-judgment. A 69-season baseball pilot tests when an individual’s own record becomes more predictive than their reference class, with careful caveats for hiring. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #predictive-modeling, #sports-analytics, #referential-evaluation, #lahman-baseball-database, #reference-class-prediction, #empirical-bayes, #irep, #resume-screening, and more. This story was written by: @elodieaishwarya. Learn more about this writer by checking @elodieaishwarya's about page, and for more stories, please visit hackernoon.com. The article uses public baseball data to test when an individual’s prior record beats reference-class prediction. It argues that hiring may need a similar shift away from static résumé categories, while clearly stating that the baseball number does not directly transfer to hiring.
September 10 · 8 min
Knowledge Graphs: What They Are, Where They Live, and How to Build One
This story was originally published on HackerNoon at: https://hackernoon.com/knowledge-graphs-what-they-are-where-they-live-and-how-to-build-one. A beginner-friendly guide to knowledge graphs covering triples, nodes, edges, graph databases, real-world uses, and Python examples. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #knowledge-graphs, #knowledge-graph, #graph-databases, #python-knowledge-graph, #networkx-tutorial, #knowledge-graph-python, #rdf-triplestore, #graph-data-modeling, and more. This story was written by: @subhash1986. Learn more about this writer by checking @subhash1986's about page, and for more stories, please visit hackernoon.com. A beginner-friendly guide to knowledge graphs covering triples, nodes, edges, graph databases, real-world uses, and Python examples.
September 3 · 9 min
I Built an AI Job Tracker While Job Hunting: The AI Was the Easy Part
This story was originally published on HackerNoon at: https://hackernoon.com/i-built-an-ai-job-tracker-while-job-hunting-the-ai-was-the-easy-part. A backend engineer's honest build story: aggregating 11 UK job-board APIs into one schema, with AI scoring and cover letters on top. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #python-programming, #fastapi, #artificial-intelligence, #job-search, #full-stack-development, #ai-job-tracker, #apis, #side-project, and more. This story was written by: @bogdus1k. Learn more about this writer by checking @bogdus1k's about page, and for more stories, please visit hackernoon.com. I built SearchWork - a tool that searches 11 job boards at once, scores each role against my CV, and generates cover letters. The AI was the easy part; normalising 11 messy job APIs into one schema was the real work.
September 2 · 8 min
AI Agents Are Opening Up, but What About the Data?
This story was originally published on HackerNoon at: https://hackernoon.com/ai-agents-are-opening-up-but-what-about-the-data. A2A and MCP are making AI agents easier to connect, but proprietary data remains harder to move, govern, and secure across enterprise AI systems. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #enterprise-data, #enterprise-ai, #data-interoperability, #data-portability, #ai-agent-permissions, #data-governance, #multi-agent-data-access, #good-company, and more. This story was written by: @jonstojanjournalist. Learn more about this writer by checking @jonstojanjournalist's about page, and for more stories, please visit hackernoon.com. A2A and MCP are making it easier for AI agents built by different providers to communicate and share tools. But agent interoperability doesn't solve the harder data problem. This article explores why proprietary data platforms, duplicated pipelines, fragmented permissions, and multi-agent access controls could limit enterprise AI flexibility even as open protocols make the agent layer more portable.
September 2 · 5 min
Scaling a Kafka Consumer From 4K to 25K Events per Second While Preserving Ordering
This story was originally published on HackerNoon at: https://hackernoon.com/scaling-a-kafka-consumer-from-4k-to-25k-events-per-second-while-preserving-ordering. A production redesign using batching, safe offset commits, and record-level fallback raised Kafka consumer throughput from 4K to 25K without adding partitions. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #apache-kafka, #software-architecture, #distributed-systems, #software-development, #scalability, #performance-optimization, #kafka-consumer, #event-streaming, and more. This story was written by: @manjushaguntur. Learn more about this writer by checking @manjushaguntur's about page, and for more stories, please visit hackernoon.com. A Kafka consumer processing 4,000 events per second had to meet an 8,000 event SLO. Rather than add partitions, we redesigned the consumer around batching, safe offset commits, and record-level fallback. The solution was validated in production at approximately 25,000 events per second while preserving ordering and at-least-once delivery for recoverable events. This article explains how the design handled successful batches, transient failures, and terminal failures without silently losing events or blocking a partition.
September 1 · 10 min
Modeling Valve-Closure Pressure Spikes in Miniature Fluid Systems
This story was originally published on HackerNoon at: https://hackernoon.com/modeling-valve-closure-pressure-spikes-in-miniature-fluid-systems. Estimate liquid-line pressure spikes from fast solenoid-valve closure with a simple model, Python code, limitations, and bench-test guidance. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #python-programming, #hardware-engineering, #iot, #fluid-dynamics, #ivd-analyzers, #pmic, #lab-automation, #solenoid-valves, and more. This story was written by: @alexhu-fluidics. Learn more about this writer by checking @alexhu-fluidics's about page, and for more stories, please visit hackernoon.com. A rigid-column model shows how density, liquid-column length, flow velocity, and valve closure time set the pressure-spike scale. Python makes the screening calculation repeatable, while bench testing and a transient model remain essential near component limits.
September 1 · 17 min
Fivetran vs. Apache SeaTunnel: Managed ELT or Open-Source Control?
This story was originally published on HackerNoon at: https://hackernoon.com/fivetran-vs-apache-seatunnel-managed-elt-or-open-source-control. A successful pipeline doesn’t guarantee trusted data. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-pipeline, #data-science, #data-security, #fivetran-vs-seatunnel, #apache-seatunnel, #fivetran-alternative, #managed-elt, #data-quality, and more. This story was written by: @zhoujieguang. Learn more about this writer by checking @zhoujieguang's about page, and for more stories, please visit hackernoon.com. See how to enable reliable movement, recovery, and schema evolution.
August 29 · 6 min
The Narrative Wars: Tracking Public Web Influence Operations Across the Globe
This story was originally published on HackerNoon at: https://hackernoon.com/the-narrative-wars-tracking-public-web-influence-operations-across-the-globe. NGOs use public web monitoring tools and web data analysis to reveal geopolitical influence operations in Europe and Asia. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #real-world-web-data, #digital-propaganda-networks, #geopolitical-tensions, #ai-monitoring, #information-warfare, #information-integrity, #south-korea-fimi, #hackernoon-top-story, and more. This story was written by: @cerniauskas. Learn more about this writer by checking @cerniauskas's about page, and for more stories, please visit hackernoon.com. Geopolitical conflicts are increasingly shifting to the information domain, requiring civil society organizations to have better tools for monitoring the public web. By analyzing data at scale, teams like FactCheck.LT and Doublethink Lab are able to expose influence operations ranging from large-scale state propaganda to subtle, early-stage inauthentic activities. Ultimately, closing the resource gap between state-backed actors and civil society requires a stronger coalition between technology providers and the public-interest organizations working to defend information integrity.
August 28 · 17 min
Teaching Funnels To Understand Non-Linear Customer Journeys
This story was originally published on HackerNoon at: https://hackernoon.com/teaching-funnels-to-understand-non-linear-customer-journeys. Linear funnels miss branching journeys, competing goals and abandoned paths. A graph-based framework can model customer behaviour more accurately. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #funnel-analytics, #customer-journey-analytics, #attribution-modelling, #directed-acrylic-graphs, #networkx, #yaml-configuration, #conversion-analytics, and more. This story was written by: @rahuln. Learn more about this writer by checking @rahuln's about page, and for more stories, please visit hackernoon.com. Traditional funnel analytics forces irregular customer journeys into fixed sequences. GoalFlow instead represents milestones as a graph and generates reusable pipelines from structured definitions, although its anchoring rules and route metrics need clearer and internally consistent explanations.
August 27 · 10 min
Building an AI-Powered A2P SMS Fraud Detection Platform Using XGBoost: A Machine Learning Approach
This story was originally published on HackerNoon at: https://hackernoon.com/building-an-ai-powered-a2p-sms-fraud-detection-platform-using-xgboost-a-machine-learning-approach. How explainable AI, behavioral analytics, and operational dashboards can help telecom operators detect messaging fraud & avert it in real time. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #random-forest, #xgboost-model, #machine-learning, #a2p, #fraud-detection, #ai-powered-fraud-detection, #telecom-fraud, #ai-intelligence, and more. This story was written by: @adnanmalik83. Learn more about this writer by checking @adnanmalik83's about page, and for more stories, please visit hackernoon.com. This article presents my design and implementation of an AI-powered A2P SMS Fraud Detection Platform using XGBoost, combining machine learning, explainable AI, and operational analytics.
August 26 · 9 min
Measuring Decision Confidence in Business Intelligence and Analytics
This story was originally published on HackerNoon at: https://hackernoon.com/measuring-decision-confidence-in-business-intelligence-and-analytics. Learn why Decision Confidence is the KPI organizations should measure. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #business-intelligence, #data-analytics, #data-engineering, #data-governance, #decision-confidence, #bi-dashboards, #kpi-management, #data-driven-decision-making, and more. This story was written by: @venkatasaibolineni. Learn more about this writer by checking @venkatasaibolineni's about page, and for more stories, please visit hackernoon.com. More dashboards don't create better decisions—trusted data does. Here's why Decision Confidence may be the most important analytics KPI organizations aren't measuring
August 17 · 5 min
Your Healthcare Integration Is Only as Good as Its Data Mapping Decisions
This story was originally published on HackerNoon at: https://hackernoon.com/your-healthcare-integration-is-only-as-good-as-its-data-mapping-decisions. Healthcare data mapping mistakes rarely come from bad tech. They come from mismatched assumptions between systems. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #healthcare-data-integration, #data-mapping, #clinical-data-mapping, #healthcare-interoperability, #ehr-integration, #legacy-healthcare-data, #healthcare-data-reconciliation, #semantic-interoperability, and more. This story was written by: @ubaidpisuwala. Learn more about this writer by checking @ubaidpisuwala's about page, and for more stories, please visit hackernoon.com. Data mapping failures in healthcare integrations almost never trace back to broken code. They trace back to unmapped exceptions, mismatched status definitions between systems, historical data that doesn't behave like current data, and reconciliation that gets added after the fact instead of built in from day one. Teams that treat mapping specs as living documents, not one-time deliverables, avoid most of the costly rework that shows up months after go-live.
August 13 · 13 min
Choosing a Python Sentence Boundary Detection Library
This story was originally published on HackerNoon at: https://hackernoon.com/choosing-a-python-sentence-boundary-detection-library. A high-accuracy, rule-based Sentence Boundary Detector (SBD) with a drop-in adapter for pysbd, delivering faster and more accurate segmentation. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #nlp, #natural-language-processing, #sentence-level-analysis, #tokenization, #sentence-boundary-detection, #sentence-segmentation, #pysbd-vs-yasbd-lib, #nlp-libraries, and more. This story was written by: @speedyk-005. Learn more about this writer by checking @speedyk-005's about page, and for more stories, please visit hackernoon.com. yasbd-lib is a fast, streaming-first Python sentence splitter that avoids text-mutation bugs to preserve exact spans, whereas pysbd is a mature, widely used rule-based splitter that is stable but largely unmaintained upstream.
August 4 · 6 min
This Is How Observability Starts - With Modeling Pipeline Runs
This story was originally published on HackerNoon at: https://hackernoon.com/this-is-how-observability-starts-with-modeling-pipeline-runs. Learn why modeling pipeline runs as first-class entities is the missing foundation for reliable observability in ETL and data processing systems. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #data, #observability, #python, #etl, #software-architecture, #software-engineering, #modeling-pipeline, and more. This story was written by: @emotta. Learn more about this writer by checking @emotta's about page, and for more stories, please visit hackernoon.com. Most data pipelines treat observability as an afterthought, relying on logs, metrics, and traces that lack business context. This article argues that observability starts much earlier: by explicitly modeling each pipeline run as a first-class entity.
August 4 · 12 min
Why AI-Assisted Data Engineering Needs Executable Specifications
This story was originally published on HackerNoon at: https://hackernoon.com/why-ai-assisted-data-engineering-needs-executable-specifications. Spec-Driven Data Engineering turns business rules, schemas, validation, and orchestration into versioned contracts that guide AI coding agents. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #spec-driven-development, #spec-driven-data-engineering, #executable-data-specifications, #ai-assisted-data-engineering, #data-pipeline-contracts, #versioned-business-logic, #data-pipeline-architecture, and more. This story was written by: @shuhua. Learn more about this writer by checking @shuhua's about page, and for more stories, please visit hackernoon.com. AI-assisted coding is enabling data engineers to build pipelines faster than ever, but it is also increasing platform fragmentation. As business logic, transformation rules, and architectural decisions become embedded in prompts, critical system knowledge becomes difficult to trace, validate, and maintain. This article introduces Spec-Driven Data Engineering (SDDE), an approach that treats executable specifications as the source of truth for data platforms. By moving system knowledge from temporary prompts into versioned specifications, organizations can improve consistency, governance, traceability, and reuse while allowing AI coding agents to generate and evolve data pipelines at scale.

Showing 1–20 of 36 episodes