Lesson 1: Core data concepts
Exam DP-900 checks that you understand basic data ideas and Microsoft cloud data services. Microsoft's study guide has four skill areas, and this course has one lesson for each.
| Skill area | Weight |
|---|---|
| Describe core data concepts | 25–30% |
| Identify considerations for relational data on Azure | 20–25% |
| Describe considerations for working with non-relational data on Azure | 15–20% |
| Describe an analytics workload | 25–30% |
Fundamentals exams last 45 minutes. A passing score is 700 or greater on a scale of 1 to 1,000. It is a scaled score, so 700 may not mean 70% of the points.
Three ways to represent data
Data usually describes entities, the things an organization cares about, such as customers and products. Each entity has attributes, such as a customer's name, address and phone number.
- Structured data follows a fixed schema, so every record has the same fields. It is usually stored in tables: one row per entity and one column per attribute.
- Semi-structured data has some structure but allows variation. For example, some customers have two email addresses and some have none. JSON is a common format.
- Unstructured data has no specific structure: documents, images, audio, video and other binary files.
Common file formats
| Format | What it is |
|---|---|
| Delimited text (CSV) | Plain text. Commas separate fields and a new line ends each row. Human-readable. |
| JSON | Objects in braces and collections in square brackets, with name: value pairs. Good for structured and semi-structured data. |
| XML | Tags in angle brackets. Popular in the 1990s and 2000s, now largely replaced by the less wordy JSON. |
| BLOB | Raw binary data, such as images, video and audio. |
| Avro | Row-based (from Apache). A JSON header describes the binary records. |
| Parquet | Columnar (from Apache). Data for each column is stored together in row groups. |
| Delta Lake | Builds on Parquet by adding a transaction log for ACID transactions and versioning. |
Databases
Relational databases store structured data in tables linked by keys and use SQL. Non-relational (NoSQL) databases don't use a relational schema. Four common types:
- Key-value: a unique key with a value in any format.
- Document: a key-value database where the value is a JSON document the system can query.
- Column family: rows and columns, with related columns grouped into families.
- Graph: entities are nodes, and links between them are relationships.
Transactional workloads (OLTP)
Online transactional processing (OLTP) systems record many small units of work, such as payments. They are optimized for both reads and writes, to create, read, update and delete (CRUD) records quickly. They follow ACID rules. Picture moving $100 from chequing to savings:
- Atomicity: all or nothing. If the credit to savings fails, the debit from chequing is undone too.
- Consistency: data moves only from one valid state to another.
- Isolation: transactions at the same time can't interfere, so no one sees a half-finished transfer.
- Durability: once committed, a change stays, even after a power failure and restart.
Analytical workloads
Analytical systems are read-mostly and hold large amounts of historical data. Data is extracted, transformed and loaded (ETL), or extracted and loaded first and transformed afterwards (ELT). It lands in a data lake (large volumes of files), a data warehouse (relational tables optimized for reading) or a data lakehouse, which combines the flexible storage of a data lake with the SQL querying of a warehouse. Data may then be aggregated in a semantic model (once called a cube) for reports and dashboards.
Roles
| Role | Main responsibilities |
|---|---|
| Database administrator | Database availability, performance, backup and recovery plans, and granting or denying user access |
| Data engineer | Data ingestion pipelines, cleansing and transformation, analytical data stores, and monitoring data pipelines |
| Data analyst | Exploring data for trends, building analytical models, and creating reports and visualizations |
On Azure, common stores are Azure SQL and open-source databases for relational data, Azure Storage and Azure Cosmos DB for non-relational data, and Microsoft Fabric or Azure Databricks for analytics. Lessons 2–4 cover them.
Practical exercise: sort a company's data
A security company keeps this data. Classify each item.
| Item | Description |
|---|---|
| 1 | Client invoices, with the same columns for every invoice |
| 2 | JSON guard profiles; some list two certifications, some none |
| 3 | Body-camera video files |
| 4 | The billing app that records each payment as it happens |
| 5 | A dashboard of hours worked per site over three years |
Which items are structured, semi-structured and unstructured?
Show answer
Item 1 is structured (fixed schema). Item 2 is semi-structured (JSON with variation). Item 3 is unstructured (video).
Are items 4 and 5 transactional or analytical?
Show answer
Item 4 is transactional (OLTP): many small writes as they happen. Item 5 is analytical: aggregated historical data.
A payment is saved, then the power fails. After the restart the payment is still there. Which ACID property is this?
Show answer
Durability: a committed transaction stays committed.
Lesson 2: Relational data on Azure
In a relational database, each table holds one kind of entity. Each row is one instance, such as one customer. Every row has the same columns, each with a data type (text, decimal, integer, date). A column with no value holds NULL, for example MiddleName for someone with no middle name.
Normalization and keys
Normalization is a design process that reduces duplicate data and enforces data integrity. Put each entity in its own table and each attribute in its own column, give each row a primary key, and link tables with foreign keys.
- A primary key uniquely identifies each row.
- A foreign key stores the primary key of a related entity, such as CustomerID in an Order table. The customer's address is then stored only once.
- A composite key combines columns, such as OrderNo plus ItemNo.
- Referential integrity means a foreign key must match an existing primary key, so you can't save an order for a customer who doesn't exist.
SQL statements
SQL is the standard language for relational databases. Dialects include Transact-SQL (T-SQL) for SQL Server and Azure SQL, pgSQL for PostgreSQL and PL/SQL for Oracle.
| Group | Statements | Used to |
|---|---|---|
| Data Definition Language (DDL) | CREATE, ALTER, DROP, RENAME | Create, change and remove objects |
| Data Control Language (DCL) | GRANT, DENY, REVOKE | Manage permissions |
| Data Manipulation Language (DML) | SELECT, INSERT, UPDATE, DELETE | Read, add, change and delete rows |
SELECT, UPDATE and DELETE apply to every row unless a WHERE clause limits them, and AND means both conditions must be true. SQL has no "are you sure?" prompt, so UPDATE or DELETE without WHERE changes or removes all rows. JOIN links rows from two tables, usually by matching a foreign key to a primary key.
Other database objects
- View: a virtual table based on a SELECT query. You query it much like a table.
- Stored procedure: saved SQL statements that run on command, with parameters so the same logic works for different values.
- Index: helps find rows fast, like a book's index. But it uses storage and must be updated whenever rows are inserted, updated or deleted.
The Azure SQL family
| Service | Type | Key points |
|---|---|---|
| SQL Server on Azure Virtual Machines | IaaS | Full SQL Server and full control. You manage the operating system, updates and backups. Good for lift-and-shift with no changes. |
| Azure SQL Managed Instance | PaaS | A full SQL Server instance with many databases and near-100% compatibility. Backups and patching are automated. Good for most migrations. |
| Azure SQL Database | PaaS | A fully managed database with most core database-level features of SQL Server. Good for new cloud apps. |
Azure SQL Database can be a single database (including a serverless option that scales automatically) or an elastic pool, where several databases share resources. A pool suits databases that are busy at different times.
Open-source databases
- Azure Database for MySQL: a PaaS version of MySQL Community Edition, the leading open-source database for LAMP (Linux, Apache, MySQL, PHP) apps.
- Azure Database for PostgreSQL: PostgreSQL is a hybrid relational-object database that can store custom data types and geometric data.
- Both let organizations move apps to Azure without major changes. For MySQL, Flexible Server is the recommended deployment option for new workloads.
Use this Product table to practise:
| A | B | C | D | |
|---|---|---|---|---|
| 1 | ID | Name | Price | Category |
| 2 | 1 | Hammer | 12.50 | Tools |
| 3 | 2 | Drill | 89.99 | Tools |
| 4 | 3 | Tape | 4.25 | Supplies |
Practical exercise: normalize a work order sheet
A cleaning company tracks work orders in one sheet.
| A | B | C | D | E | F | |
|---|---|---|---|---|---|---|
| 1 | OrderNo | Client | ClientAddress | Service | Rate | Hours |
| 2 | 101 | Bow River Dental | 12 Main St, Calgary | Floor wax | 45.00 | 3 |
| 3 | 101 | Bow River Dental | 12 Main St, Calgary | Window wash | 38.00 | 2 |
| 4 | 102 | Prairie Books | 88 King Ave, Red Deer | Floor wax | 45.00 | 4 |
| 5 | 103 | Bow River Dental | 12 Main St, Calgary | Carpet clean | 52.00 | 1 |
What data is repeated?
Show answer
Each client's name and address repeat on every row, and each service's rate repeats every time it is used.
Which tables would a normalized design use?
Show answer
Client, Service, WorkOrder and WorkOrderLine, so each entity has its own table.
What key could identify each WorkOrderLine row?
Show answer
A composite key of OrderNo plus a line number. OrderNo is also a foreign key to WorkOrder.
The company wants to move its on-premises SQL Server instance and all its databases to Azure with almost no changes, without managing VMs. Which service fits?
Show answer
Azure SQL Managed Instance: near-100% compatible, with automated backups and patching.
Lesson 3: Non-relational data on Azure
Azure Blob Storage
Azure Blob Storage stores large amounts of unstructured data as blobs (binary large objects). Blobs live in containers in a storage account, and you control read and write access at the container level.
| Blob type | Use |
|---|---|
| Block blob | Large files that change rarely |
| Page blob | Fixed 512-byte pages for random reads and writes, such as virtual machine disks |
| Append blob | Data added only at the end, such as logs |
| Access tier | Use it for |
|---|---|
| Hot (default) | Data accessed often |
| Cool | Data accessed less often (keep 30+ days) |
| Cold | Data accessed even less often than Cool (keep 90+ days) |
| Archive | Rarely needed history. Lowest storage cost; offline; retrieval can take up to 15 hours (keep 180+ days) |
To read an archived blob, change its tier to Hot, Cool or Cold and wait until it is rehydrated. A lifecycle management policy can move blobs to cheaper tiers as they age and delete old blobs.
Data Lake Storage Gen2 and OneLake
Azure Data Lake Storage Gen2 adds a hierarchical file system to Azure Storage for analytics tools such as Azure Databricks. You turn on the hierarchical namespace option of a storage account. Upgrading an existing account is one-way.
Microsoft OneLake is one logical data lake for the whole organization. It comes with every Microsoft Fabric tenant and is built on ADLS Gen2.
Azure Files
Azure Files hosts cloud-based network file shares in a storage account, like the shares many offices run on their own servers. SMB works with Windows, Linux and macOS. NFS is for Linux only and needs the SSD tier and a virtual network.
Azure Table storage
Azure Table storage is a NoSQL store of key/value rows. Apart from the keys and a timestamp, columns can vary. There are no foreign keys or relationships, so data is usually denormalized: each row holds a whole entity.
- Each row's unique key is a partition key plus a row key.
- Rows with the same partition key are stored together. Searching by partition key reduces reads and writes (I/O), so queries are faster.
- For new workloads, Azure Cosmos DB for Table is recommended.
Azure Cosmos DB
Azure Cosmos DB is a fully managed NoSQL database. Items in a container don't need the same structure, and a partition key spreads the data out.
- Global: users read from the nearest region and, with multi-region writes turned on, also write to the nearest region.
- Fast: reads and writes under 10 milliseconds at the 99th percentile.
- Automatic indexing of all properties by default.
- Use cases: IoT telemetry, gaming profiles and leaderboards, retail catalogues and carts, web and mobile apps.
- For complex multi-table joins, Azure SQL Database is better suited.
| Cosmos DB API | Data and query | Choose it when |
|---|---|---|
| NoSQL (formerly SQL API) | JSON documents; SQL-like queries | Building a new app (recommended) |
| MongoDB | BSON; MongoDB Query Language | Migrating a MongoDB app |
| Table | Key-value rows | Key-value storage at scale |
| Apache Cassandra | Column-family tables; CQL | Migrating a Cassandra workload |
| Apache Gremlin | Graphs of vertices and edges | Relationships matter, such as recommendations or fraud detection |
Practical exercise: choose a storage option
Pick a service, tier or API for each need of a property management company.
Seven years of old tenancy records that are almost never opened.
Show answer
Blob Storage in the Archive tier: lowest storage cost, but retrieval can take hours.
Shared office folders used on Windows and Mac laptops.
Show answer
Azure Files with SMB.
A new mobile app with flexible JSON tenant profiles in many countries.
Show answer
Azure Cosmos DB for NoSQL: JSON, global distribution, recommended for new apps.
Application logs that only get new lines at the end.
Show answer
An append blob.
Lesson 4: Analytics workloads
Layers of large-scale analytics
- Ingestion and processing: data from transactional stores, files and streams is loaded into a data lake or warehouse, usually with ETL or ELT.
- Analytical data store: a data warehouse, a data lake or a lakehouse.
- Analytical data model: a semantic model of tables, relationships, hierarchies and measures.
- Data visualization: reports and dashboards.
Analytical data stores
- A data warehouse often uses a star schema: fact tables hold numeric values such as sales, and dimension tables hold entities to group by, such as customer, product and time. A snowflake schema adds tables linked to dimensions, such as Category linked to Product.
- A data lake is a file store that uses schema-on-read: the schema is applied when data is read, not when it is stored.
- A lakehouse exposes files in a lake as SQL tables. Delta Lake adds relational features on top of Parquet and is the standard lakehouse format for Microsoft Fabric and Azure Databricks.
Microsoft Fabric and Azure Databricks
Microsoft Fabric is a unified software as a service (SaaS) analytics platform built on OneLake. It brings data engineering, warehousing, real-time analytics, data science and Power BI into one workspace.
- Lakehouse: Delta tables with Spark notebooks and a SQL endpoint.
- Warehouse: a SQL Server-compatible relational warehouse for structured data, SQL teams and strong schema enforcement.
- Data Factory: pipelines for moving and transforming data.
- Mirroring: continuously copies databases such as Azure SQL Database into OneLake in near real time, without pipelines.
- Shortcuts: live references to external data, with no copying.
Azure Databricks is built on Apache Spark, runs as a managed service in your Azure subscription and uses Delta Lake as its native format.
Batch and streaming data
Batch processing collects records and processes them together, like a monthly credit card bill. Stream processing handles each event as it arrives.
| Batch | Stream | |
|---|---|---|
| Data scope | The whole dataset | Recent data or a rolling time window |
| Latency | Typically hours | Seconds or milliseconds |
| Analysis | Complex analytics | Simple aggregates, such as rolling averages |
Real-time analytics services
- Azure Event Hubs ingests high volumes of events. Azure IoT Hub is similar but optimized for IoT devices.
- Azure Stream Analytics: PaaS streaming jobs that read a source, apply a perpetual query and write to an output.
- Fabric Real-Time Intelligence: Eventstreams ingest data, an Eventhouse stores event data queried with KQL (Kusto Query Language), and Activator triggers actions when data meets conditions.
Power BI
Analysts build a model and report in Power BI Desktop, publish it to the Power BI service to share it and schedule refreshes, and users view it in a browser or the phone app.
A semantic model has measures (numbers such as revenue) and dimensions (such as product and time). Hierarchies let users drill from year to month to day.
| Visual | Best for |
|---|---|
| Card | One key figure |
| Bar or column chart | Comparing categories |
| Line chart | Trends over time |
| Pie chart | Parts of a whole |
| Scatter plot | The relationship between two numeric measures |
| Map | Comparing values by location |
Practical exercise: plan an analytics report
A parking company wants a report and live alerts.
Which visual fits: (1) revenue for each of 24 months, (2) revenue across 8 lots, (3) each lot's share of the total?
Show answer
(1) Line chart for a trend over time. (2) Bar or column chart to compare categories. (3) Pie chart for parts of a whole.
Staff need an alert within seconds when a gate sensor reports a fault. Batch or stream?
Show answer
Stream processing, because each event must be handled as it arrives.
Which Fabric Real-Time Intelligence item could send the alert?
Show answer
Activator, which triggers actions when data meets set conditions.
Sources
- Microsoft Learn: Study guide for Exam DP-900
- Microsoft Learn: Exam DP-900
- Microsoft Learn: Microsoft Certified: Azure Data Fundamentals
- Microsoft Learn: Exam scoring and score reports
- Microsoft Learn: Exam duration and exam experience
- Microsoft Learn: Renew your Microsoft Certification
- Microsoft Learn: Course DP-900T00 Introduction to Microsoft Azure Data
- Microsoft Learn module: Explore core data concepts
- Microsoft Learn module: Explore data roles and services
- Microsoft Learn module: Explore fundamental relational data concepts
- Microsoft Learn module: Explore relational database services in Azure
- Microsoft Learn module: Explore Azure Storage for nonrelational data
- Microsoft Learn module: Explore fundamentals of Azure Cosmos DB
- Microsoft Learn module: Explore fundamentals of large-scale analytics
- Microsoft Learn module: Explore fundamentals of real-time analytics
- Microsoft Learn module: Explore fundamentals of data visualization
- Microsoft Learn: Introduction to Azure Blob Storage