Data Pipelines Using Azure Data Factory

Leveraging cloud services makes us easy to do integration part of on-premise data sources with cloud services, ETL (Extract, Transform, and Load) has been a part of every digital transformation project that we undertake as…

Apache Spark Do’s and Don’ts

One of the main tools for every data engineer tools box would be apache-spark, knowing how to use it effectively and efficiently will pay him dividends in long run. There are many different tools in…

ETL-Extract

The first part of an ETL process involves extracting the data from the source systems. In many cases this is the most challenging aspect of ETL, as extracting data correctly will set the stage for…