Data Pipelines Using Azure Data Factory

Leveraging cloud services makes us easy to do integration part of on-premise data sources with cloud services, ETL (Extract, Transform, and Load) has been a part of every digital transformation project that we undertake as…

Apache Spark Do’s and Don’ts

One of the main tools for every data engineer tools box would be apache-spark, knowing how to use it effectively and efficiently will pay him dividends in long run. There are many different tools in…

ETL-Extract

The first part of an ETL process involves extracting the data from the source systems. In many cases this is the most challenging aspect of ETL, as extracting data correctly will set the stage for…

B-Tree on Big Data

One can say that with a strong conviction that most of the commercial success of MySQL comes from the InnoDB engine. Oracle acquired InnoDB several years before MySQL!I believe that Oracle acquired not only the…

Training the Ability to find solution

As a programmer, technical knowledge is most important, but in addition to improving technical ability to tackle a problem, there are other aspects to consider most. One of the important skills to get is Structured…