I'm using python beam on google dataflow, my pipeline looks like this: Read image urls from file >> Download images >> Process images The problem is that I can't let Download images step scale...

I'm trying this example for retrieve data from GCP Pub/Sub at DataFlow. import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.IOException; import java.time.Instant; i...

I was reading the docs about SCHEMAS in Apache BEAM but i can not understand what its purpose is, how and why or in which cases should i need to use them. What is the difference between using schem...

I have a luigi task that performs some non-stable computations. Think of an optimization process that sometimes does not converge. import luigi MyOptimizer(luigi.Task): input_param: luigi.Param...

I want to paint a diagram where you can see the dataflow of a java program, and if there are one or multiple threads handling the data. Sequence charts don't show multithreading and get very confu...

I have an SSIS package with a Data Flow that takes an ADO.NET data source (just a small table), executes a select * query, and outputs the query results to a flat file (I've also tried just pulling...

I try to set up controller service account for Dataflow. In my dataflow options I have: options.setGcpCredential(GoogleCredentials.fromStream( new FileInputStream("key.json")).createScop...

i am attempting to run a Dataflow job using Flex template, but i am getting stuck on a 'module not found error' and i cannot figure out why so here is the structure of my directory |__ modules |__...

Scenario - Running Dataflow jobs on project A using a shared VPC to use the region and subnetwork of host project B On the service account, I have following permission on both project A and B Compu...

I have a TransformManyBlock with the following design: Input: Path to a file Output: IEnumerable of the file's contents, one line at a time I am running this block on a huge file (61GB), which ...

Is it possible to get TransformManyBlocks to send intermediate results as they are created to the next step instead if waiting for the entire IEnumerable<T> to be filled? All testing I've d...

I have a requirement to make a scalable process. The process has mainly I/O operations with some minor CPU operations (mainly deserializing strings). The process query the database for a list of ur...

I'm doing a simple pipeline using Apache Beam in python (on GCP Dataflow) to read from PubSub and write on Big Query but can't handle exceptions on pipeline to create alternatives flows. On a sim...

I'm trying to launch a Dataflow job on GCP using Apache Beam 0.6.0. I am compiling an uber jar using the shade plugin because I cannot launch the job using "mvn:execjava". I'm including this depend...

What is the difference between Data Flow Analysis and Abstract Interpretation and are they used for the same purpose? What are the pros and cons of these two relative to each other.

We are currently in the process of deploying a new spring data flow stream application in our aws EKS cluster. As part of this, the pods launched by the skipper should have the IAM roles defined in...

I'm very newby with GCP and dataflow. However , I would like to start to test and deploy few flows harnessing dataflow on GCP. According to the documentation and everything around dataflow is imper...

I have a dataflow pipeline in which I consume pubsub messages, treat them, and then publish to pubsub. Whenever I have too many calculations (ie I increase the amount of treatment for each message...

I have a general question on side inputs and broadcasting in the context of Apache Beam. Does any additional variables, lists, maps that are need for computation during processElement, need to be p...

To my knowledge, I should be able to use EnvironmentObject to observe & access model data from any view in the hierarchy. I have a view like this, where I display a list from an array that's in...

We are currently working on a streaming pipeline on Apache Beam with DataflowRunner. We are reading messages from Pub/Sub and do some processing on them and afterwards we window them in slidings wi...

I have a use case when I need to capture the data flow from one API to another. For example my code reads data from database using hibernate and during the data processing I convert one POJO to ano...

I'm setting up a slow-changing lookup Map in my Apache-Beam pipeline. It continuously updates the lookup map. For each key in lookup map, I retrieve the latest value in the global window with acc...

Say in Dataflow/Apache Beam program, I am trying to read table which has data that is exponentially growing. I want to improve the performance of the read. BigQueryIO.Read.from("projectid:dataset....

My current understanding is that NiFi processor properties are specific to that processor. So adding a new property to a processor will only be visible within that processor and not be passed on to...