Topic outline

  • WELCOME LETTER

    Dear Students

    Welcome to the Big Data and IoT Theory (CSE 412) &  Lab (CSE 413) courses, I,   Abdul Hye Zebon will be your course teacher in this online journey of learning.

    I care about your success in these courses. I'm glad you are here.

    Md. Abdul Hye Zebon
    Lecturer,
    Department of Computer Science and Engineering,
    Daffodil International University



    Course Instructor



    Name

    Md. Abdul Hye Zebon
    Designation

    Lecturer, Daffodil International University
    Room Number

    807-A (AB4)
    Email

    zebon.cse0399.c@diu.edu.bd
    Contact No

    +880-1521438443


    Course Rationals

    Big Data refers to a massive set of data that no conventional data management tool can handle. Big Data is therefore a concept that allows access to gigantic databases in real time. Big Data's main objectives are to improve a company's or system's responsiveness to a large amount of data collected, increase productivity and refine knowledge of customer behavior, so that it can offer personalized offers or advertisements and create new trends.

    The Internet of Things (IoT) is a concept that connects physical or virtual objects to the internet. The technology very often used is the sensor, allowing to link a physical object such as a watch, a drone or even a speaker, to the internet. If for a long time the few objects connected to the Internet were the telephone and the computer, this is no longer the case today and every year new types of objects incorporating IOT technology are born. IOT is one of the greatest technological revolutions of our era and its potential for exploitation is immense. IOT could have a huge impact on the cars of the future or on the new versions of smart-cities, an urban space connected to the Internet, thus significantly improving the lives of users, while reducing the negative impact of these on the planet. 


    Course Objective

    To provide a solid conceptual understanding of the fundamentals of Big Data and IoT. More specifically,

    • To learn the basic concepts of Big Data and IoT.
    • To learn the architecture of Hadoop Cluster and Ecosystem.
    • To learn distributed data storage and processing systems.
    • To learn IoT Levels.
    • To learn the concepts of Communication Protocols of IoT.
    • To learn Communication Models of IoTs.
    • To learn the fundamental tools used in Big Data.



    Course Learning Outcome (CLO): (at the end of the course, students will be able to do:)

    CLO1            Interpret the components, tools, and techniques of Big Data.                                                                                                            
    CLO2 Classify the Hadoop model and identify their differences in implementation.
    CLO3 Explain how information can be sent via IoT communication protocols and Models.
    CLO4 Determine the various IoT levels and their applications and Fields in development sectors.


    Assessment Plan

    Final Exam
    40
    Mid-term Exam
    25
    3 Class Tests
    15
    Attendance
    07
    Assignment
    05
    Presentation
    08
    Total
    100

    • WEEK 1: INTRODUCTION

      • Topic of Discussion

        O1
        Introduction with the students
        O2
        Introduction to Data and Information
        O3
        Small Data and Big Data
        O4
        Types of Big Data
        O5
        Big Data Characteristics

        Expected Learning Outcome

        O1
        Learn about data and information
        O2
        Learn about many types of data
        O3
        Explore several characteristics of big data

      • Resources of Learning

        01
        Introduction to Big Data
        02
        Reference Video

    • Lab WEEK 1 : INTRODUCTION

      • Topic of Discussion

        O1
        Introduction to Big Data
        O2
        Introduction to Python
        O3
        Introduction to Apache Spark

        Expected Learning Outcome

        O1
        Install PyCharm
        O2
        Work with basic python programming
        O3
        Install Apache Spark

        Resources of Learning

        01
        Install Pycharm on Ubuntu
        02
        Install Pycharm on Windows
        03
        Install Apache Spark on Ubuntu
        04
        Install Apache Spark on Windows
        05
        Install Apache Spark + Python = PySpark with Jupyter Notebook on Windows

    •  WEEK 2 : BIG DATA ANALYTICS

      • Topic of Discussion

        O1
        Why big data analytics?
        O2
        What is big data analytics?
        O3
        Lifecycle of big data analytics
        O4
        Types of big data analytics
        O5
        Tools used in big data analytics

        Expected Learning Outcome

        O1
        Learn about the big data analytics
        O2
        Learn about the application of big data analytics.
        O3
        Learn about the big data application domains

        Resources of Learning

        01
        Big Data Analytics Slide

    • LAB WEEK 2 : INSTALLTION OF PYSPARK

      • Topic of Discussion

        O1
        Installation of PySpark
        O2
        Introduction to PySpark

        Expected Learning Outcome

        O1
        Complete Setup of PySpark
        O2
        Knowledge about PySpark programming

        Resources of Learning

        01
        PySpark with Jupyter Notebook on Ubuntu
        02
        PySpark Programming Example
        03
        Python Basics Notebooks

    • WEEK 3 : INTRODUCTION TO HADOOP

      • Topic of Discussion

        O1
        What is Hadoop
        O2
        Hadoop Components
        O3
        HDFS
        O4
        Master and Slave Node
        O5
        Map Reduce and Yarn

        Expected Learning Outcome

        O1
        Understand Hadoop
        O2
        Understanding HDFS
        O3
        Understanding Map Reduce and Yarn

      • Resources of Learning

        01
        Introduction to Hadoop

    • LAB WEEK 3 : INTRODUCTION TO PYSPARK PROGRAMMING

      • Topic of Discussion

        O1
        Basic programming with pyspark
        O2
        Basic Programming with RDD

        Expected Learning Outcome

        O1
        To Work with RDD transformations and action
        O2
        Working with pyspark programming

        Resources of Learning

        01
        Basics of RDD files

    • WEEK 4 : MAP REDUCE AND YARN

      • Topic of Discussion

        O1
        Map Reduce
        O2
        YARN
        O3
        YARN with Map Reduce
        O4
        Hadoop MapReduce
        O5
        Hadoop MapReduce Example

        Expected Learning Outcome

        O1
        Know details about MapReduce and YARN
        O2
        Differences between Hadoop v1.0 and v2.0
        O3
        Using MapReduce in word count

      • Resources of Learning

        01
        Map Reduce and YARN

    • LAB WEEK 4 : INTRODUCTION TO RDD AND DATAFRAME

      • Topic of Discussion

        O1
        Introduction to Dataframes
        O2
        WordCount through RDDs
        O3
        Project Related Discussion

        Expected Learning Outcome

        O1
        Work with dataset
        O2
        To update the project

        Resources of Learning

        01
        Working With DataFrame Files

    • WEEK 5 : APACHE SQOOP, HIVE AND PIG

      • Topic of Discussion

        O1
        Hadoop Ecosystem
        O2
        Apache Sqoop
        O3
        Apache Hive
        O4
        Apache Pig
        O5
        Differences between Pig and Hive
        O6
        The architecture of Sqoop, Pig, and Hive

        Expected Learning Outcome

        O1
        Details about apache sqoop, hive, and pig
        O2
        Learn about the differences between them.

        Resources of Learning

        01
        Apache Pig, Sqoop and Hive

    • LAB WEEK 5: DATAFRAMES AND SPARK SQL

      • Topic of Discussion

        O1
        Creating data frames
        O2
        Loading and transforming data through different sources
        O3
        Covid- 19 example dataset

        Expected Learning Outcome

        O1
        Able to work on different dataset.
        O2
        Work with database query

        Resources of Learning

        01
        Project Task Description
        02
        DataFrame Files

    • WEEK 6 : MIDTERM


      • Mid Syllabus: W1-W5

        1. Introduction to Big Data

        2. Big Data Analytics

        3. Introduction to Hadoop

        4. Map Reduce and Yarn

        5. Hadoop Cluster and Ecosystem


    • WEEK 6 : HADOOP CLUSTER AND ECOSYSTEM

      • Topic of Discussion

        O1
        Hadoop Cluster
        O2
        Hadoop Ecosystem
        O3
        Benefits of Hadoop Cluster
        O4
        Challenges of Hadoop Cluster

        Expected Learning Outcome

        O1
        Know details about Hadoop Cluster and Ecosystem
        O2
        Know about different types of tools.

      • Resources of Learning

        01
        Hadoop Cluster
        02
        Hadoop Ecosystem

    • LAB WEEK 6 : Linear Regression

      • Topic of Discussion

        O1
        What is a Regression Problem?
        O2
        When should we use Linear Regression Algorithm?
        O3
        How to pre-process data for training the model?
        O4
        How to train a Linear Regression model in Pyspark?

        Expected Learning Outcome

        O1
        Able to Train Linear Regression Model in Pyspark

        Resources of Learning

        01
        Linear Regression Files

    • WEEK 7: INTRODUCTION TO IOT

      • Topic of Discussion

        O1
        Introduction to IoT
        O2
        Domain Specific IoTs
        O3
        IoT applications for industry

        Expected Learning Outcome

        O1
        Know Details about IoT and its importance.
        O2
        Know Details about IoT domains and its applications.

        Resources of Learning

        01
        Introduction to IoT

    • ASSIGNMENT

      • Your Assignment will contain 5 points regarding the algorithm that have been assigned to you. No word limit or page limit has been set. You can submit either a hand written assignment or a printed copy. The points that you have to include in your assignment are,

        • Introduction
        • Working Procedure of the Algorithm
        • Advantages
        • Disadvantages
        • Real Life Problem Solving using the Algorithm (Example)
        Best of luck.

      • Students of Section 53-A, Please find your assigned topic from the google sheet below.
        Google Sheet Link: Click Here
        Restricted Not available unless: You belong to Section - A

      • Students of Section 53-B, Please find your assigned topic from the google sheet below.
        Google Sheet Link: Click Here

        Restricted Not available unless: You belong to Section - B

      • Students of Section 53-C, Please find your assigned topic from the google sheet below.
        Google Sheet Link: Click Here

        Restricted Not available unless: You belong to Section - C

      • Students of Section 53-D, please provide your assigned topic in the google sheet below. Those who haven't got any topic previously, leave the space against your ID for me to assign a topic.
        Google Sheet Link: Click Here

        Restricted Not available unless: You belong to Section - D

    • LAB WEEK 7 : Decision Tree


      • Topic of Discussion

        O1
        What is a Classification Problem?
        O2
        When should we use Decision Tree Algorithm?
        O3
        How to pre-process data for training the model?
        O4
        How to train a Decision Tree model in Pyspark?

        Expected Learning Outcome

        O1
        Able to Train Decision Tree Model in Pyspark

        Resources of Learning

        01
        Decision Tree Files

    • WEEK 8: INTRODUCTION TO IOT (Part 2)


      • Topic of Discussion

        O1
        Introduction to IoT
        O2
        Domain Specific IoTs
        O3
        IoT applications for industry

        Expected Learning Outcome

        O1
        Know Details about IoT and its importance.
        O2
        Know Details about IoT domains and its applications.

        Resources of Learning

        01
        Introduction to IoT Part 2

    • LAB WEEK 8 : IoT LAB

      • Topic of Discussion

        O1
        Basic Concept of IoT
        O2
        Work with Arduino
        O3
        Work with different sensors

        Expected Learning Outcome

        O1
        Able to work using Arduino.
        O2
        Able to work with different sensors

        Resources of Learning

        01
        Online Simulator
        02
        Tinkercad Design and codes

    • WEEK 9 : M2M AND IOT


      • Topic of Discussion

        O1
        Introduction to IoT
        O2
        Domain-Specific IoTs
        O3
        IoT applications for industry

        Expected Learning Outcome

        O1
        Know Details about IoT and its importance.
        O2
        Know Details about IoT domains and their applications.

        Resources of Learning

        01
        M2M and IoT_MMH
        03
        Recorded Class Video

    • Week 14: Final Exam


      • Final Exam Syllabus:

        1. Hadoop MapReduce and Yarn

        2. Big Data Analytics

        3. Introduction to IoT-

        4. IoT_part 2

        5. IoT and M2M