I need to process some values in a data pipeline and need to use the value later somewhere in the program.
Here is a simple example
import apache_beam as beam
p = beam.Pipeline()
resu=(
p
| beam.Create([1,3,5,3,5,3])
| beam.CombineGlobally(beam.combiners.MeanCombineFn())
| beam.io.WriteToText("result.txt")
)
p.run()
Now the mean value is calculated and put into the file "result.txt". If I need to use the mean value later in the program I need to do a file io operation. I want to have the result come in memory as a variable instead. How do I achieve this?
something like
mean_value=resu.values()
# use mean_value as a regular variable
some_other_value=mean_value/2
beam.pvalue.AsSingleton
to provide the value as side input to anotherMap/ParDo
Refer: link – Heedful