Integrating Apache Nifi with Azure Storage

 In this post we will see how Apache Nifi can be used to handle the blobs/files in Azure Blob storage.

We will go through:

• Creating a Storage account in Microsoft Azure
• Creating Nifi template that download the files/blobs from Azure blob storage
• Creating a Nifi template that upload files to the blob storage
• Creating a Nifi template that delete blobs/files from the blob storage

Prerequisites:

It is expected that, you have fundamental knowledge on 

• Microsoft Azure
• Cloud computing
• Apache Nifi
• Data pipeline

Steps:

1. Create the Storage Account

1. Go to Azure home portal: https://portal.azure.com/?quickstart=True#home 

2. Click on Storage Account :

3. Click on Add Button, if you don’t have a Storage account. Below figures are for your reference:

4. Now go to the Storage Account portal: https://portal.azure.com/?quickstart=True#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts 

5. Click on the storage account name

6.  Under “Blob service” section

1. Click on Containers.

2. Click + Container button to create a new container

3. Give a name and leave others to default

4. Now click on the Create button at the bottom. 

5. Now click on container name 

6. Click on Upload button to upload a file

7. Now you need to get the Access Keys

1. Under Settings section, click on Access Keys 

2. Now click on Show Keys button

3. Note down the Key (not connection string) under key1

2. Nifi template for downloading the blobs from a container

8. Go to Nifi interface

9. Create a ListAzureBlobStorage processor.

10. Configure with following properties

1. Container Name: Get the name you have created in previous step

2. Storage Credentials:

1. Create a new Controller Services “AzureStorageCredentialsControllerService”

2. Properties are: Storage Account name (same as you have created before), Storage Account Key (same as you have noted down in previous step)

3. Now Enable the Controller Service 

11. Create a FetchAzureBlobStorage processor with the same properties of  ListAzureBlobStorage processor. 

12. Create a PutFile processor, with the following properties:

1. Directory: /tmp/azure/success (you can change this according to your requirement)

2. Leave rest properties to default

13. Create a PutFile processor, with the following properties. This is just to track if anything goes wrong:

1. Directory: /tmp/azure/fail (you can change this according to your requirement)

2. Leave rest properties to default

14. Now the Nifi template should look like below:

3. Nifi template to UPLOAD files to a container

15. Get the following processors and connect them.

16. Properties of GetFile:

17. Properties of PutAzureBlobStorage processor:

4. Nifi template to delete files/blob from a container

18. Create the following nifi processor:

19. Properties of ListAzureBlobStorage processor:

20. Properties of DeleteAzureBlobStorageprocessor:

Here I am deleting only one blob(i.e. Image 2.png) . But you can leave the Blob properties to its default value to delete all the blobs