Course Details
Topic 1: Introduction to data engineering on Azure
- What is data engineering
- Important data engineering concepts
- Data engineering in Microsoft Azure
Topic 2: Introduction to Azure Data Lake Storage Gen2
- Understand Azure Data Lake Storage Gen
- Enable Azure Data Lake Storage Gen2 in Azure Storage
- Compare Azure Data Lake Store to Azure Blob storage
- Understand the stages for processing big data
- Use Azure Data Lake Storage Gen2 in data analytics workloads
Topic 3: Introduction to Azure Synapse Analytics
- What is Azure Synapse Analytics
- How Azure Synapse Analytics works
- When to use Azure Synapse Analytics
- Exercise - Explore Azure Synapse Analytics
Topic 4: Use Azure Synapse serverless SQL pool to query files in a data lake
- Understand Azure Synapse serverless SQL pool capabilities and use cases
- Query files using a serverless SQL pool
- Create external database objects
- Exercise - Query files using a serverless SQL pool
Topic 5: Use Azure Synapse serverless SQL pools to transform data in a data lake
- Transform data files with the CREATE EXTERNAL TABLE AS SELECT statement
- Encapsulate data transformations in a stored procedure
- Include a data transformation stored procedure in a pipeline
- Exercise - Transform files using a serverless SQL pool
Topic 6: Create a lake database in Azure Synapse Analytics
- Understand lake database concepts
- Explore database templates
- Create a lake database
- Use a lake database
- Exercise - Analyze data in a lake database
Topic 7: Secure data and manage users in Azure Synapse serverless SQL pools
- Choose an authentication method in Azure Synapse serverless SQL pools
- Manage users in Azure Synapse serverless SQL pools
- Manage user permissions in Azure Synapse serverless SQL pools
Topic 8: Analyze data with Apache Spark in Azure Synapse Analytics
- Get to know Apache Spark
- Use Spark in Azure Synapse Analytics
- Analyze data with Spark
- Visualize data with Spark
- Exercise - Analyze data with Spark
Topic 9: Transform data with Spark in Azure Synapse Analytics
- Modify and save dataframes
- Partition data files
- Transform data with SQL
- Exercise: Transform data with Spark in Azure Synapse Analytics
Topic 10: Use Delta Lake in Azure Synapse Analytics
- Understand Delta Lake
- Create Delta Lake tables
- Create catalog tables
- Use Delta Lake with streaming data
- Use Delta Lake in a SQL pool
- Exercise - Use Delta Lake in Azure Synapse Analytics
Topic 11: Analyze data in a relational data warehouse
- Design a data warehouse schema
- Create data warehouse tables
- Load data warehouse tables
- Query a data warehouse
- Exercise - Explore a data warehouse
Topic 12: Load data into a relational data warehouse
- Load staging tables
- Load dimension tables
- Load time dimension tables
- Load slowly changing dimensions
- Load fact tables
- Perform post load optimization
- Exercise - load data into a relational data warehouse
Topic 13: Manage and monitor data warehouse activities in Azure Synapse Analytics
- Scale compute resources in Azure Synapse Analytics
- Pause compute in Azure Synapse Analytics
- Manage workloads in Azure Synapse Analytics
- Use Azure Advisor to review recommendations
- Use dynamic management views to identify and troubleshoot query performance
Topic 14: Secure a data warehouse in Azure Synapse Analytics
- Understand network security options for Azure Synapse Analytics
- Configure Conditional Access
- Configure authentication
- Manage authorization through column and row level security
- Exercise - Manage authorization through column and row level security
- Manage sensitive data with Dynamic Data Masking
- Implement encryption in Azure Synapse Analytics
Topic 15: Build a data pipeline in Azure Synapse Analytics
- Understand pipelines in Azure Synapse Analytics
- Create a pipeline in Azure Synapse Studio
- Define data flows
- Run a pipeline
- Exercise - Build a data pipeline in Azure Synapse Analytics
Topic 16: Use Spark Notebooks in an Azure Synapse Pipeline
- Understand Synapse Notebooks and Pipelines
- Use a Synapse notebook activity in a pipeline
- Use parameters in a notebook
- Exercise - Use an Apache Spark notebook in a pipeline
Topic 17: Plan hybrid transactional and analytical processing using Azure Synapse Analytics
- Understand hybrid transactional and analytical processing patterns
- Describe Azure Synapse Link
Topic 18: Implement Azure Synapse Link with Azure Cosmos DB
- Enable Cosmos DB account to use Azure Synapse Link
- Create an analytical store enabled container
- Create a linked service for Cosmos DB
- Query Cosmos DB data with Spark
- Query Cosmos DB with Synapse SQL
- Exercise - Implement Azure Synapse Link for Cosmos DB
Topic 19: Implement Azure Synapse Link for SQL
- What is Azure Synapse Link for SQL?
- Configure Azure Synapse Link for Azure SQL Database
- Configure Azure Synapse Link for SQL Server
- Exercise - Implement Azure Synapse Link for SQL
Topic 20: Get started with Azure Stream Analytics
- Understand data streams
- Understand event processing
- Understand window functions
- Exercise - Get started with Azure Stream Analytics
Topic 21: Ingest streaming data using Azure Stream Analytics and Azure Synapse Analytics
- Stream ingestion scenarios
- Configure inputs and outputs
- Define a query to select, filter, and aggregate data
- Run a job to ingest data
- Exercise - Ingest streaming data into Azure Synapse Analytics
Topic 22: Visualize real-time data with Azure Stream Analytics and Power BI
- Use a Power BI output in Azure Stream Analytics
- Create a query for real-time visualization
- Create real-time data visualizations in Power BI
- Exercise - Create a real-time data visualization
Topic 23: Introduction to Microsoft Purview
- What is Microsoft Purview?
- How Microsoft Purview works
- When to use Microsoft Purview
Topic 24: Discover trusted data using Microsoft Purview
- Search for assets
- Browse assets
- Use assets with Power BI
- Integrate with Azure Synapse Analytics
Topic 25: Catalog data artifacts by using Microsoft Purview
- Register and scan data
- Classify and label data
- Search the data catalog
Topic 26: Manage Power BI assets by using Microsoft Purview
- Register and scan a Power BI tenant
- Search and browse Power BI assets
- View Power BI metadata and lineage
Topic 27: Integrate Microsoft Purview and Azure Synapse Analytics
- Catalog Azure Synapse Analytics data assets in Microsoft Purview
- Connect Microsoft Purview to an Azure Synapse Analytics workspace
- Search a Purview catalog in Synapse Studio
- Track data lineage in pipelines
- Exercise - Integrate Azure Synapse Analytics and Microsoft Purview
Topic 28: Explore Azure Databricks
- Get started with Azure Databricks
- Identify Azure Databricks workloads
- Understand key concepts
- Exercise - Explore Azure Databricks
Topic 29: Use Apache Spark in Azure Databricks
- Get to know Spark
- Create a Spark cluster
- Use Spark in notebooks
- Use Spark to work with data files
- Visualize data
- Exercise - Use Spark in Azure Databricks
Topic 30: Use Delta Lake in Azure Databricks
- Get Started with Delta Lake
- Create Delta Lake tables
- Create and query catalog tables
- Use Delta Lake for streaming data
- Exercise - Use Delta Lake in Azure Databricks
Topic 31: Use SQL Warehouses in Azure Databricks
- Get started with SQL Warehouses
- Create databases and tables
- Create queries and dashboards
- Exercise - Use a SQL Warehouse in Azure Databricks
Topic 32: Run Azure Databricks Notebooks with Azure Data Factory
- Understand Azure Databricks notebooks and pipelines
- Create a linked service for Azure Databricks
- Use a Notebook activity in a pipeline
- Use parameters in a notebook
- Exercise - Run an Azure Databricks Notebook with Azure Data Factory
Final Assessment
- Written Assessment - Short Answer Questions (WA-SAQ)
- Practical Performance (PP)
Course Info
Promotion Code
Promo or discount cannot be applied to WSQ courses
Minimum Entry Requirement
Knowledge and Skills
- Able to operate using computer functions with minimum Computer Literacy Level 2 based on ICAS Computer Skills Assessment Framework
- Minimum 3 GCE ‘O’ Levels Passes including English or WPL Level 5 (Average of Reading, Listening, Speaking & Writing Scores)
Attitude
- Positive Learning Attitude
- Enthusiastic Learner
Experience
- Minimum of 1 year of working experience.
Target Year Group : 21-65 years old
Minimum Software/Hardware Requirement
Software:
You need to sign up a Azure account (Credit Card is required).
Hardware: Windows and Mac Laptops
About Progressive Wage Model (PWM)
The Progressive Wage Model (PWM) helps to increase wages of workers through upgrading skills and improving productivity.
Employers must ensure that their Singapore citizen and PR workers meet the PWM training requirements of attaining at least 1 Workforce Skills Qualification (WSQ) Statement of Attainment, out of the list of approved WSQ training modules.
For more information on PWM, please visit MOM site.
Funding Eligility Criteria
Individual Sponsored Trainee | Employer Sponsored Trainee |
|
|
SkillsFuture Credit:
PSEA:
|
Absentee Payroll (AP) Funding:
SFEC:
|
Steps to Apply Skills Future Claim
- The staff will send you an invoice with the fee breakdown.
- Login to the MySkillsFuture portal, select the course you’re enrolling on and enter the course date and schedule.
- Enter the course fee payable by you (including GST) and enter the amount of credit to claim.
- Upload your invoice and click ‘Submit’
SkillsFuture Level-Up Program
The SkillsFuture Level-Up Programme provides greater structural support for mid-career Singaporeans aged 40 years and above to pursue a substantive skills reboot and stay relevant in a changing economy. For more information, visit SkillsFuture Level-Up Programme
Get Additional Course Fee Support Up to $500 under UTAP
The Union Training Assistance Programme (UTAP) is a training benefit provided to NTUC Union Members with an objective of encouraging them to upgrade with skills training. It is provided to minimize the training cost. If you are a NTUC Union Member then you can get 50% funding (capped at $500 per year) under Union Training Assistance Programme (UTAP).
For more information visit NTUC U Portal – Union Training Assistance Program (UTAP)
Steps to Apply UTAP
- Log in to your U Portal account to submit your UTAP application upon completion of the course.
Note
- SSG subsidy is available for Singapore Citizens, Permanent Residents, and Corporates.
- All Singaporeans aged 25 and above can use their SkillsFuture Credit to pay. For more details, visit www.skillsfuture.gov.sg/credit
- An unfunded course fee can be claimed via SkillsFuture Credit or paid in cash.
- UTAP funding for NTUC Union Members is capped at $250 for 39 years and below and at $500 for 40 years and above.
- UTAP support amount will be paid to training provider first and claimed after end of class by learner.
Job Roles
- Data Engineer
- Azure Data Engineer
- Cloud Data Engineer
- Business Intelligence Developer
- Data Architect
- Data Scientist
- Machine Learning Engineer
- Big Data Engineer
- Data Analyst
- Database Administrator
- Cloud Solutions Architect
- DevOps Engineer
- Software Developer
- IT Professional
- Systems Analyst
Trainers
Solomon Soh Zhe Hong: Solomon is ACTA certified and has trained and coached over 100 professionals in the area of data science, python programming and coding. Solomon is a Certified AI Engineer Associate by AI Singapore and holds certifications in Alibaba Cloud Architect and Alteryx respectively. Solomon interests include Reinforcement Learning, Natural Language Processing and Time-Series analysis.
Dr. Alfred Ang: Dr. Alfred Ang is the founder of Tertiary Courses. He is a serial entrepreneur. He founded OSWeb2Design Singapore Pte Ltd in 2007 offering web development, e-commerce store development, graphics design, ebook publishing, mobile apps development, and digital marketing services. He established the first online gardening store in Singapore, Eco City Hydroponics Pte Ltd in 2000, offering a wide range of gardening products such as seeds, plant nutrients, hydroponics kits etc. Eco City Hydroponics has become the most popular and successful gardening store in Singapore. He founded Tertiary Infotech Pte Ltd in 2012 and transformed the business to a training platform, Tertiary Courses in 2014. Tertiary Courses offers a wide range of SkillsFuture courses for PMETs to upgrade their skills and knowledge. He also established Tertiary Courses Malaysia in 2016. He also founded Tertiary Robotics in 2015 offering Arduino, Raspberry Pi, Microbit and Robotics products
Dr. Alfred Ang earned his Ph.D. from National University of Singapore in 2000, majoring in Electrical and Electronics Engineering. He also completed an online MBA course with U21 Global based in Australia. He obtained his B.Sc (Hons) from National University of Singapore in 1992, majoring in Physics. He topped his Physics cohort for 3 consecutive years and funded his degree study with Book price, awards and tuition. He has worked in Defence, Electronics and Semiconductor Industries. His current interests include Machine Learning, Deep Learning, Artificial Intelligence, Internet of Things, Robotics and Programming.
Dr. Alfred Ang is a ACTA certified trainer and DACE certified course developer. He was Distinguished Toastmasters (DTM) and Senior Member of IEEE. He has published more than 20 peer reviewed papers and co-inventors for more than 20 inventions.