I am writing an application that plots financial data and interacts with a realtime feed of such data. Due to the nature of the task, live market data may be received very frequently in one-trade-a...

Questions How much more expensive is it to load small files (eg. 4K) using Snowpipe than say 16K, 500K or 1-10Mb (the recommended file size). Note: This question implies it is more expensive to l...

I am creating a data ingestion pipeline using Cloud Run. My Cloud Run api gets called everytime a file is dropped in a GCS bucket via Pub Sub. I need to load some metadata that contains text for th...

Because it's so easy to install on Debian stable, I decided to use PostgreSQL 9.6 to build a datawarehouse for some data I need to process. The first step is to load the data into the database with...

I’m investigating feasability of sending spring boot application logs directly into elastic search. Without using filebeats or logstash. I believe the Ingest plugin may help with this. My initial ...

This is my first time here, so sorry if I don't post fine, and sorry for my bad English. I'm trying to configure Apache Flume and Elasticsearch sinks. Everything is ok, it seems that it runs fine,...

I'm trying to put into simple terms when spark pulls data through the driver, and then when spark doesn't need to pull data through the driver. I have 3 questions - Let's day you have a 20 TB ...