Anix Lynchgozeroshot.dev·Dec 6, 2024Python Automation #3: Data ingestion w/pandas, SQLAlchemy, dataprep.connector, petl1. Read Data from CSV, Excel, or JSON (pandas.read_csv, pandas.read_excel, pandas.read_json) Read CSV: import pandas as pd # Read data from CSV df = pd.read_csv("data.csv") print(df.head()) Output (Sample): ID Name Age 0 1 Alice 25 1 2 ...dataprep.connector
Muhammad Fahad Bashirmfahadbashir.hashnode.dev·Sep 12, 20245.Vector Stores: Efficient Storage and Retrieval for EmbeddingsIn this continuation of our series of Retrieval-Augmented Generation (RAG), we will learn about the final step of the ingestion pipeline—vector stores. Previously, we covered embeddings in detail, from understanding what they are to implementing them...12 likes·57 readsImplementing RAG systems from Scratch in-depthTutorial
Christopher Wilsontechshopper.hashnode.dev·Jul 17, 2024Advanced Python Programming Techniques: Working with Big DataIn today's data-driven world, the ability to process and analyze large volumes of data is an invaluable skill. Python, with its robust ecosystem of libraries and tools, is a powerful ally in the realm of big data. This article explores advanced Pytho...Advanced Python
Abhishek Jaiswaldataplumbing.hashnode.dev·Jul 3, 2024Data Engineering with Large Language Models:In recent years, data engineering has emerged as a critical field, driving the backbone of data-centric organizations. The advent of Large Language Models (LLMs) has further transformed this domain, introducing new paradigms for managing, processing,...data-engineering
Cloud Tunedcloudtuned.hashnode.dev·May 21, 2024Unleash the Power of Data Prepper: Simplifying Data Ingestion and TransformationUnleash the Power of Data Prepper: Simplifying Data Ingestion and Transformation OpenSearch Data Prepper is a powerful tool designed to streamline the process of ingesting, transforming, and preparing data for analysis in OpenSearch (formerly known a...data-prepper
ByteHousebytehouse.hashnode.dev·Dec 12, 2023The Modern Data Stack - An essential guideModern Data Stack. Sorry, what? So, everyone and their pet have a tech stack. Folks in the data world have ‘modern data stacks’. But, what exactly does that mean? A modern data stack (MDS) refers to a set of technologies and tools that organisations ...Modern data stack
Warui Wanjiruwaruithemystery.hashnode.dev·Dec 5, 2023Navigating the Data Ingestion Landscape: A Comprehensive Guide, Part 1Introduction In the realm of data engineering, the process of data ingestion plays a pivotal role in the journey of information from source systems to storage. This comprehensive guide aims to delve into the intricacies of data ingestion, exploring k...42 readsData Ingestiondata solutions architecture
Karl Bolingerkbolinger.hashnode.dev·Apr 14, 2023Data ingestion using REST APIsIn today's world, data is being generated at an unprecedented rate. Big and small organizations are collecting data from various sources such as websites, social media, and IoT devices. The data needs to be collected, processed, and analyzed to deriv...113 readsData Engineering Basicsdata-engineering
