Currently I'm using an AWS Glue job to load data into RedShift, but after that load I need to run some data cleansing tasks probably using an AWS Lambda function. Is there any way to trigger a Lamb...

I'm setting up a new Jupyter Notebook in AWS Glue as a dev endpoint in order to test out some code for running an ETL script. So far I created a basic ETL script using AWS Glue but, for some reason...

I have a source as XML and has a huge number of records. just for the sample I have pasted 1 record below : <?xml version='1.0' encoding='UTF-8'?><wd:Report_Data xmlns:wd="urn:com.wor...

I have a CSV file with a {LF} delimiting each row and a date column with the date format as "12/20/2010" (including quotation marks) My destination column is a SQL Server 2008 database table of ty...

I try to use configs in dag using "trigger w/config". def execute(**kwargs): dag_run = kwargs['dag_run'] start_date = dag_run.conf['start_dt'] if 'start_dt' in dag_run.conf.keys() el...

I have the following task to solve: Files are being sent at irregular times through an endpoint and stored locally. I need to trigger a DAG run for each of these files. For each file the same ta...

I used the code below to fill a data table - OleDbDataAdapter oleDA = new OleDbDataAdapter(); DataTable dt = new DataTable(); oleDA.Fill(dt, Dts.Variables["My_Result_Set"].Value); I get the er...

I'm trying to write a simple SCDF flow that reads from Kafka, filters the messages by presence of specific value and pushes data into Mongo. As part of this i had to wrote following #jsonPath #jso...

I'm looking into ETL tools (like Talend) and investigating whether Apache Nifi could be used. Could Nifi be used to perform the following: Pick up two CSV files that are placed on local disk Join ...

I have downloaded and installed Visual Studio 2022. Then after click on modify Now, I want to create SSIS package, for this I have started VS22 and in "manage extensions" when I t...

I have problems with SSIS process(acctually the same problem occurs for two different processes). So, we are doing some ETL work using SSIS. We have Business Intelligence project that executes with...

I have a whole bunch of data in AWS S3 stored in JSON format. It looks like this: s3://my-bucket/store-1/20190101/sales.json s3://my-bucket/store-1/20190102/sales.json s3://my-bucket/store-1/20190...

I am using SQL Server 2014 with SSIS I have a data set like this: ID Name Status 1 Awesome "Store" Active 2 Market, Place Active 3 Vendor Active In SSMS, when the results are in the grid and I ...

I want to use ETL to read data from S3. Since with ETL jobs I can set DPU to hopefully speed things up. But how do I do it? I tried import sys from awsglue.transforms import * from awsglue.util...

On a SQL 2016 Server I have a job that calls an SSIS package. That package is in a project in the SSISDB and has parameters. One of those parameters is a string type that is blank as a default. I ...

I'm trying to execute a stored procedure via SSIS Error: 0xC0207014 at PO Header, OLE DB Source [59]: The SQL command requires a parameter named "@SessionID", which is not found in the paramet...

We are designing an Big data solution for one of our dashboard applications and seriously considering Glue for our initial ETL. Currently Glue supports JDBC and S3 as the target but our downstream ...

I am trying to make Microsoft.Azure.Services.AppAuthentication and its dependencies work with SSIS script task. How do I resolve assembly reference errors? static ScriptMain() { AppDomain.Curren...

I have a Data Flow Task that is hanging on excecution. The flow is simple, makes two queries to different tables (Both with a couple of joins), then sorts and merges the otuputs through a common i...

I have a job and multiple transformations. If i wanted to define a database connection in the Job , and use the same database connection for all the transformations , how do I go about it ? I am us...

Am currently building an ETL pipeline, which outputs tables of data (order of ~100+ GBs) to a downstream interactive dashboard, which allows filtering the data dynamically (based on pre-defin...

I tried to search posts, but I only found solutions for SQL Server/Access. I need a solution in MySQL (5.X). I have a table (called history) with 3 columns: hostid, itemname, itemvalue. If I do a s...

I'm setting up Airflow right now and loving it, except for the fact that my dags are perpetually running behind. See the picture below - this was taken on 2/19 at 15:50 UTC, and you can see that fo...

We currently use Datastage ETL to - Export a CSV/text file with data from 15 tables(3 different schemas) on a daily basis. I am wondering If there is a simpler way to accomplish this with out usin...

I have fetched the data as List < T > by reading from different formats e.g. CSV, Parquet, Avro, JSON. I want to validate the data with mostly feature e.g. The temperature should remain with ...