i am working on a project where i utilize log4j2 logging. while developing in intellij, all works fine and the logging is done as expected. the log4j2.xml is linked through java property passed to...

Here are the steps to the current process: Flafka writes logs to a 'landing zone' on HDFS. A job, scheduled by Oozie, copies complete files from the landing zone to a staging area. The staging da...

Using Hbase API (Get/Put) or HBQL API, is it possible to retrieve timestamp of a particular column?

With: Java 1.8.0_231 Hadoop 3.2.1 Flume 1.8.0 Have created a hdfs service on 9000 port. jps: 11688 DataNode 10120 Jps 11465 NameNode 11964 SecondaryNameNode 12621 NodeManager 12239 ResourceMa...

Am new in the Apache Flume. I need to install the flume on top of the HDFS cluster environment. I did Google it, all are saying using the cloudera distribution but I need to install and confi...

I have a simple flume setup with a HTTP souce and a sink that writes the POST request payload to a file. (This complete setup is on a Linux machine). After that my task is to do a performance test...

I have a cluster of Kafka with 5 brokers and I'm using Consul Service Discovery to put their IPs into a dns record. kafka.service.domain.cc A 1.1.1.1 2.2.2.2 ... 5.5.5.5 Is it recommended to use...

I am trying to use Flume Appender Properties of log4j2 .But the following errors are obtained when run the program . 2016-01-20 16:36:42,436 main ERROR Appenders contains an invalid element or at...

I have built an api web application, which is published on IIS Server, I am trying to configure Apache Flume to listen that web api and to save the response of http petitions in HDFS, this is the p...

I am trying to configure flume with Hive to save flume output to hive table with Hive Sink type. I have single node cluster. I use mapr hadoop distribution. Here is my flume.conf agent1.sources =...

I have a simple Flume agent with the following configuration: agent.sources = http-source agent.sinks = logger-sink agent.channels = logger-channel # HTTP Source ############################### ...

I am trying to read messages on Kafka topic, but I am unable to read it. The process gets killed after sometime, without reading any messages. Here is the rebalancing error which I get: [2014-03-...

I have server X that has Hadoop and Flume installed, and I have server Y that has neither but is on the same network. Server Y currently stores data into a log file that is continuously written tw...

Both Flume and Sqoop are meant for data movement, then what is the difference between them? Under what condition should I use Flume or Sqoop?

What is difference between Apache flume and Apache storm? Is is possible to ingest logs data into Hadoop cluster using storm? Both are used for streaming data so can storm be used as an alternat...

I am a newbie to Flume and Hadoop. We are developing a BI module where we can store all the logs from different servers in HDFS. For this I am using Flume. I just started trying it out. Succesfull...

This is my first time here, so sorry if I don't post fine, and sorry for my bad English. I'm trying to configure Apache Flume and Elasticsearch sinks. Everything is ok, it seems that it runs fine,...

what are best practices for "importing" streamed data from Kafka into HBase? The usecase is as follows: Vehicle sensor data are streamed to Kafka. Afterwards, these sensordata must be transformed ...

I am using the HTTPSource in Flume for receiving POST events in json format as follows: {"username":"xyz","password":"123"} My question is: Do I have to modify the source of the events (I mean t...

I need to read a stream of data from some source (in my case it's UDP stream, but it shouldn't matter), transform the each record and write it to the HDFS. Is there any difference between using Fl...

Firstly I was thinking what to use to get events into Hadoop, where they will be stored and periodically analysis would be performed on them (possibly using Ooozie to schedule periodic analysis) Ka...

Not able to read and parse the File created by streaming twitter data using Flume twitter agent, neither using Java nor Avro Tools. My requirement is to convert the avro format into JSON format. W...

I have configured flume agent for my application, where source is Spooldir and sink is HDFS I am able to collect files in hdfs. agent configuration is: agent.sources = src-1 agent.channels = c1 ...

I installed Flume 1.4.0-cdh4.7.0 in CentOS (cloudera VM) I ran the following command to start the flume flume-ng agent -n agent-name -c conf -f conf/flume.conf -Dflume.root.looger=DEBUG,console but...

I am now using a CDH-5.3.1 cluster with three zookeeper instances located in three ips: 133.0.127.40 n1 133.0.127.42 n2 133.0.127.44 n3 Everything works fine when it starts, but these days I not...