Overview Setting up and configuring a Spark environment is not an easy task. The complexity is multi-fold when Spark has to be integrated with Jupyter Notebooks, Hive and HDFS. This guide simplifies the journey to setup and configure the environment on Mac OS. System Prerequisites Laptop or desktop with i3 quad-core processor or…

Introduction At present, about 2.5 quintillion bytes (2500 PetaBytes) of data is produced by humans every day (Source: Social Media Today). Processing this much quantity of data is a headache. This is where Big Data Processing comes into the play and acts as a painkiller for this headache. There are many…

Introduction In the previous blog we explored the Star & Snowflake schemas. Today, we will explore the Azure Cosmos DB Service . Within Cosmos DB there are various services such as Core (SQL), MongoDB, Cassandra, Azure Table, Gremlin (Graph). In this blog we would explore the Cassandra API service. Learning Objectives What is…

Introduction In the previous blog we have explored the Azure SQL basics. Today, we would explore how we can implement Star and Snowflake schemas in Azure SQL. Star and Snowflake schemas are the most commonly used schemas when it comes to the database models. Both have their pros and cons. When…

Introduction In this blog we will be exploring Azure SQL and its various features such as downloading queries, exporting data, uploading scripts, and much more. We would use the sample data that is already available within Azure SQL to speed up our learning curve and focus on the functionalities. Learning Objectives What is…

Introduction Extract, Transform, and Load (ETL) jobs are very common in Enterprise projects. They allow businesses to gather data from multiple sources, transform it and consolidate it to a target system. Performing ETL on cloud can be much more effective. So, let’s understand how we can implement the ETL concepts using…

Introduction Data Lakes built using Hadoop framework were lacking a very basic functionality i.e. ACID compliance. Hive tried to overcome some of the limitations by providing update functionality but the overall process was messy. Databricks (the company behind Spark) came up with a unique solution i.e. Delta Lake. Delta Lake enables…

Introduction With the accumulation of data into the Databases, over time the velocity of data generation is also increasing day-by-day. As we were struggling to deal with the volume of the data. The new obstacle was emerging in front of us i.e. Streaming Data. Now dealing with streaming data was a…

Introduction Out of the 3 major cloud providers i.e. GCP, AWS, and Azure. AWS is the oldest player in this game and the most trusted one. It has well equipped infrastructure to support Big Data, Apps, Machine Learning etc. workloads. Now let’s explore implementing Big Data Framework on AWS i.e. …