I am trying to run a Kafka Streams application in kubernetes. When I launch the pod I get the following exception: Exception in thread "streams-pipe-e19c2d9a-d403-4944-8d26-0ef27ed5c057-StreamThre...

I installed docker on Ubuntu with snap (snappy?), and then I ran this: ln -sf /usr/bin/snap /usr/local/bin/docker when I run docker build I get: unable to prepare context: unable to evaluate s...

I want to install parquet for python using pip within an Anaconda 2 installation on Windows 10. While installing I ran into the error that is described here, the installer can't find snappy-c.h. ...

I have compressed a file using python-snappy and put it in my hdfs store. I am now trying to read it in like so but I get the following traceback. I can't find an example of how to read the file in...

I have just extracted and setup spark 1.6.0 into environment that has a fresh install of hadoop 2.6.0 and hive 0.14. I have verified that hive, beeline and mapreduce works fine on examples. Howev...

On a very simple Kafka consumer app from tutorial website: https://www.baeldung.com/spring-kafka But once containerized with openJDK 17, this issue is 100% reprodicible: ERROR 1 --- [ntainer#0-0-C-...

I am trying to find a working example of how to use the remote write receiver in Prometheus. Link : https://prometheus.io/docs/prometheus/latest/querying/api/#remote-write-receiver I am able to sen...

Is it possible to use Pandas' DataFrame.to_parquet functionality to split writing into multiple files of some approximate desired size? I have a very large DataFrame (100M x 100), and am using df.t...

I'm building a cdc pipeline to read mysql binlog through maxwell and putting them into kafka my compression type is snappy in maxwell config.But at consumer end in my spring project I'm getting thi...

I am running a few tests on the storage formats available with Hive and using Parquet and ORC as major options. I included ORC once with default compression and once with Snappy. I have read many ...

How can I open a .snappy.parquet file in python 3.5? So far, I used this code: import numpy import pyarrow filename = "/Users/T/Desktop/data.snappy.parquet" df = pyarrow.parquet.read_table(filen...

Commmunity! Please help me understand how to get better compression ratio with Spark? Let me describe case: I have dataset, let's call it product on HDFS which was imported using Sqoop ImportTo...

I'm having trouble finding a library that allows Parquet files to be written using Python. Bonus points if I can use Snappy or a similar compression mechanism in conjunction with it. Thus far the ...

I'm storing files on HDFS in Snappy compression format. I'd like to be able to examine these files on my local Linux file system to make sure that the Hadoop process that created them has performed...

I need to select a compression algorithm when configuring "well-known application". Also, as part of my day job, my company is developing distributed application that deal with a fa...

I have a spark job that writes data to parquet files with snappy compression. One of the columns in parquet is a repeated INT64. When upgrading from spark 2.2 with parquet 1.8.2 to spark 3.1.1 with...

I have been using the latest R arrow package (arrow_2.0.0.20201106) that supports reading and writing from AWS S3 directly (which is awesome). I don't seem to have issues when I write and read my o...

I am trying to use fastparquet to open a file, but I get the error: RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'UNCOMPRESSED'] I have the following installed and have ...

I'm using snaps in ubuntu 20.04. Every so often, even several times a day all of a sudden my computer freezes as the memory usage foes to 100% with all available memory taken by snapd. I tried kill...

After applying sortWithinPartitions to a df and writing the output to a table I'm getting a result I'm not sure how to interpret. df .select($"type", $"id", $"time") ....

I would like to have the page number in the footer of every page generated with Snappy and Wkhtmltopdf, but i haven't found any clue about it. I can set a footer text (with options 'footer-center'...

I decided to use Parquet as storage format for hive tables and before I actually implement it in my cluster, I decided to run some tests. Surprisingly, Parquet was slower in my tests as against the...

I am trying to use Spark SQL to write parquet file. By default Spark SQL supports gzip, but it also supports other compression formats like snappy and lzo. What is the difference between these comp...

According to this Cloudera post, Snappy IS splittable. For MapReduce, if you need your compressed data to be splittable, BZip2, LZO, and Snappy formats are splittable, but GZip is not. Splittabi...

I have a large file of size 500 mb to compress in a minute with the best possible compression ratio. I have found out these algorithms to be suitable for my use. lz4 lz4_hc snappy quicklz blosc ...