NewsTech News

Data Science Tech Brief By HackerNoon

HackerNoon

Learn the latest data science updates in the tech world.

Play

Open in

28 episodes
Avg 10 min
English

Episodes 28

Newest Oldest Longest

All8 Unplayed8 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

June 25 · 4 min
How We Built a Per-Plant CO2 Dataset for 4,551 Power Stations Worldwide
This story was originally published on HackerNoon at: https://hackernoon.com/how-we-built-a-per-plant-co2-dataset-for-4551-power-stations-worldwide. An open dataset of 4,551 power stations: measured + modelled CO2, fuel, owner, capacity and climate zone. How we built it in Python, and the honest limits. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #python, #global-energy-monitor, #greenhouse-gas-data, #carbon-accounting, #climate-analytics, #energy-infrastructure, #python-etl, and more. This story was written by: @dmytroah. Learn more about this writer by checking @dmytroah's about page, and for more stories, please visit hackernoon.com. The authors built and openly published a dataset covering 4,551 power stations worldwide, combining emissions, ownership, capacity, fuel type, and climate-zone data into a single schema. The project's central finding is that only about 15% of plant-level emissions data comes from direct measurements, while the remaining 85% relies on modelled estimates, making provenance and transparency critical for anyone working with emissions datasets.
June 25 · 19 min
Eliminating Data Latency with Event-Driven Pipelines at Enterprise Scale
This story was originally published on HackerNoon at: https://hackernoon.com/eliminating-data-latency-with-event-driven-pipelines-at-enterprise-scale. How event-driven data pipelines reduce latency, automate schema changes, and improve reliability across large-scale data platforms. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #event-driven-architecture, #aws-glue, #schema-evolution, #cloud-infrastructure, #aws-step-functions, #incremental-data-processing, #hackernoon-top-story, and more. This story was written by: @rohitnagpal92. Learn more about this writer by checking @rohitnagpal92's about page, and for more stories, please visit hackernoon.com. Traditional batch-first data pipelines introduce artificial delays in data availability, forcing enterprise decisions to be made on stale information. This article introduces three production-proven event-driven architecture patterns: incremental processing of cloud data at petabyte scale, dynamic schema evolution with AStep Functions orchestration, and automated data quality reconciliation. These patterns eliminate data latency, cut infrastructure costs by as much as 85%, and enable real-time data availability for downstream analytics.
June 23 · 6 min
Scaling Self-Service Analytics in Regulated Banking With Metadata-Driven Design
This story was originally published on HackerNoon at: https://hackernoon.com/scaling-self-service-analytics-in-regulated-banking-with-metadata-driven-design. Scaling self-serve analytics in regulated banking is hard. Learn how metadata-driven design enforces governance while letting teams explore data safely Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #bigquery, #gcp, #data-governance, #mlops, #cross-cloud-data-platform, #cloud-data-engineering, #self-service-analytics, and more. This story was written by: @jeevanreddygeeredd. Learn more about this writer by checking @jeevanreddygeeredd's about page, and for more stories, please visit hackernoon.com. Self-service analytics in banking is not primarily a technology challenge. It's a governance challenge. This article explores the design of a metadata-driven analytics platform on GCP that enabled business teams to access trusted financial data without creating new silos. Key lessons include treating lineage as a first-class feature, using semantic layers to enforce consistent business logic, and prioritizing auditability over raw performance in regulated environments.
June 23 · 8 min
How to Rotate Proxies Without Breaking Login Sessions
This story was originally published on HackerNoon at: https://hackernoon.com/how-to-rotate-proxies-without-breaking-login-sessions. Learn how to rotate proxies safely without breaking login sessions, triggering CAPTCHA, or causing account verification issues. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #web-scraping, #proxy-rotation, #selenium, #browser-fingerprinting, #data-engineering, #anti-bot-detection, #cookie-management, #user-agent-rotation, and more. This story was written by: @marae. Learn more about this writer by checking @marae's about page, and for more stories, please visit hackernoon.com. Rotating proxies during an active login session can trigger logouts, CAPTCHA checks, verification prompts, or account locks. The safer approach is to keep one proxy, cookie jar, browser profile, user-agent, and fingerprint tied together for the full session. Rotate only after logout, task completion, or a clean session reset.
June 20 · 10 min
I Built an Open-Source Firebase Analytics Alternative Because I Hit 1M Events/Day Once Too Many
This story was originally published on HackerNoon at: https://hackernoon.com/i-built-an-open-source-firebase-analytics-alternative-because-i-hit-1m-eventsday-once-too-many. After hitting Firebase Analytics 1M events/day cap during a mobile game softlaunch, I built an open-source self-hosted analytics pipeline. Here's how. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #game-development, #analytics-pipeline, #self-hosted-analytics, #event-streaming, #event-tracking, #product-analytics, #firebase-analytics, and more. This story was written by: @rawbbit. Learn more about this writer by checking @rawbbit's about page, and for more stories, please visit hackernoon.com. A few years ago I was the data engineer on a mobile game soft launch when Firebase Analytics quietly started dropping events past its 1M/day cap. We didn't catch it for days. That experience pushed me to build Rawbbit — an open-source, Apache 2.0, self-hosted analytics pipeline that lands raw events as Parquet in your own object storage. This is the story of why hosted analytics fails at scale, why I chose NATS + Parquet + BigQuery external tables, and what I deliberately left out.
June 20 · 11 min
Your Redshift Cluster Is Probably Idle 85% of the Time — And You're Paying for All of It
This story was originally published on HackerNoon at: https://hackernoon.com/your-redshift-cluster-is-probably-idle-85percent-of-the-time-and-youre-paying-for-all-of-it. Your Redshift cluster is probably idle most of the day and billing you for all of it. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-analytics, #data-engineering, #data-management, #redshift-data-architecture, #redshift-provisioned, #serverless-rpu, #cloud-cost-optimization, #redshift-data-sharing, and more. This story was written by: @xavariannabarun. Learn more about this writer by checking @xavariannabarun's about page, and for more stories, please visit hackernoon.com. Your Redshift cluster is probably idle most of the day and billing you for all of it. Here's the SQL query, the breakeven formula, and two real production cases that show exactly when Serverless wins, when Provisioned wins, and when neither is the right answer.
June 18 · 4 min
What the Real Operating Data on AI Agents Tells Me as an Investor
This story was originally published on HackerNoon at: https://hackernoon.com/what-the-real-operating-data-on-ai-agents-tells-me-as-an-investor. Alexander Kopylkov on why AI agents are already running enterprise operations and what the production numbers tell him as an investor. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data, #ai, #ai-agents, #investing, #ai-in-business, #ai-customer-service, #ai-adoption, #ai-integration, and more. This story was written by: @alexanderkopylkov. Learn more about this writer by checking @alexanderkopylkov's about page, and for more stories, please visit hackernoon.com. Alexander Kopylkov, venture investor, finds that AI agents are already running core business functions at scale. Klarna automated 67% of its customer service with a single AI agent, saving $40 million. The remaining 33% of complex cases still required human judgment. Only 17% of companies have deployed agents so far, with 60% planning to within the next 12 months.Kopylkov sees the real investment opportunity in the governance layer that makes agents safe to operate on real business accounts, not in the agents themselves.
June 17 · 10 min
Building Data Quality Into the Pipeline Instead of Cleaning Up After It
This story was originally published on HackerNoon at: https://hackernoon.com/building-data-quality-into-the-pipeline-instead-of-cleaning-up-after-it. Data quality is a pipeline problem, not a form fix. Learn how developers can enforce quality through profiling, matching, and workflow automation at scale. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-quality, #data-engineering, #data-pipeline, #data-management, #data-validation, #data-governance, #data-profiling, #good-company, and more. This story was written by: @melissaindia. Learn more about this writer by checking @melissaindia's about page, and for more stories, please visit hackernoon.com. Bad data costs organisations millions annually and the damage rarely starts at the form level. It starts deep inside production pipelines where incorrect, duplicate, and inconsistent records silently corrupt every decision built on top of them. This article breaks down how developers can take ownership of data quality through five profiling modes, reference table management, standardization and parsing mapplets, deduplication matching, exception workflow automation, and production scheduling, covering the full pipeline from ingestion to deployment. The earlier quality is enforced, the cheaper it is to maintain.

Showing 21–28 of 28 episodes