Azure Storage Services
- Blob Storage
- Block blobs – Ordinary files up to 4.7 TB.
- Page blobs – Random access file up to 8 TB in size. Mostly VHD files that back VMs.
- Append blobs – like block blobs, but optimized for append operations.
- File Storage – Network file share that multiple VMs can share.
- Queue Storage – Store and retrieve messages.
Azure Storage Account
- General Purpose
- Standard (Magnetic disks)
- Premium (SSD disks, recommended for VMs)
- Blob Storage (Block blobs & Append blobs)
- Locally Redundant Storage (LRS: 3 copies in single data center)
- Zone Redundant Storage (ZRS)
- Geo Redundant Storage (GRS)
- Read Access – Geo Redundant Storage (RA – GRS)
Created resource group as it is required before creating storage account.
Created container inside blob service.
Created SQL Databases. Connected it using SSMS.
Create a azure data factory pipeline to read input file and transform the data using HDInsight on demand cluster and output the file. Input & Output file are used Azure Blob storage.
- Setup storage account
- Create a blob service and upload files to that service. You may want to create specific folders.
- Create data factory.
- Create a linked service to connect azure data store and deploy.
- Create input & output data set.
- Create pipeline
Create a azure data factory pipeline to copy input file data to SQL database. Input & Output are Azure Blob storage & Azure SQL.
Copy Activity & Copy Activity Wizard