Sending emails uisng Apache Nifi

In this post, we will see how one can send email to others using processors provided by Apache Nifi. 

Assume, we want to send email from sendername@gmail.com to receivername@gmail.com.

For this 

• we need to have the login credential for sendername@gmail.com email account. 
• Need to set Apache Nifi running.

In this demonstration, we will generate dummy random files and send them to receivername@gmail.com.

Steps:

1. Access Nifi user interface.
2. Bring GnerateFlowFile processor into the canvas 
3. Configure  GnerateFlowFile processor as shown in below figure.

Figure : Configuration of GenerateFlowFile processor

4. Now bring PutMail processor into the canvas
5. Configure  PutMail processor as shown in below figure.

Figure : Configuration of PutMail processor

6. Connect GnerateFlowFile to PutMail processor as shown below.

7. Now configure your gmail account and allow Nifi to login and send emails on your behalf.
8. For above, Sign in to your Gmail account
9. Now visit below link: https://myaccount.google.com/security?rapt=AEjHL4MPOlIXcaCs-jP2OdP7-Sg31QTMNhC7WbO6I08_VwUe2u5--dz6TedtBcYIYo0Ch2IfRYqErD677n9CCBrPnYgRnHcpXw
10. Make sure that Less secure app access is ON as shown in below:

11. Now start above processors and you should be receiving the dummy emails.

 

- That's all, comment if you face any difficulties.
 

Securing Apache Nifi with firewalld

In our previous post, we have learned how to secure Apache Nifi with Google's OAuth2. In that post we saw, how user can be authenticated via Google's OAuth client ID. 

In this post, we will see how the outside user can be blocked permanently from accessing Apache Nifi's UI. Here outside user mean, the user who is access UI from different machine.

For this, we will use firewalld to block the port. This can be done from local/remote terminal. But to automate the whole process, we will use Ansible.

It is assumed that, you have installed Nifi. This post will not cover how to install and configure Apache Nifi using Ansible.

 Here are the steps:

Steps:

1. Install firewall

1 2 3 4

- name: Install firewalld package: name: "firewalld" state: present

2. Enable Firewalld, you don't need to reboot the server.

5 6 7 8 9

- name: enable firewalld service: name: firewalld state: started enabled: yes

3. Now instruct Firewalld to block 8080 port permanently

10 11 12 13

- name: do not permit traffic in default zone on port 8080/tcp shell: "firewall-cmd --remove-port={{port}}/tcp --permanent" args: executable: /bin/bash

4. Now reload the Firewalld to reflect the changes

14 15 16 17

- name: reload firewall shell: "firewall-cmd --reload" args: executable: /bin/bash

Now run the Ansible playbook and verify if it working. Push a comment if you got any issue.

- That's all

Integrating Apache Nifi with Azure Storage

 In this post we will see how Apache Nifi can be used to handle the blobs/files in Azure Blob storage.

We will go through:

• Creating a Storage account in Microsoft Azure
• Creating Nifi template that download the files/blobs from Azure blob storage
• Creating a Nifi template that upload files to the blob storage
• Creating a Nifi template that delete blobs/files from the blob storage

Prerequisites:

It is expected that, you have fundamental knowledge on 

• Microsoft Azure
• Cloud computing
• Apache Nifi
• Data pipeline

Steps:

1. Create the Storage Account

1. Go to Azure home portal: https://portal.azure.com/?quickstart=True#home 

2. Click on Storage Account :

3. Click on Add Button, if you don’t have a Storage account. Below figures are for your reference:

4. Now go to the Storage Account portal: https://portal.azure.com/?quickstart=True#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts 

5. Click on the storage account name

6.  Under “Blob service” section

1. Click on Containers.

2. Click + Container button to create a new container

3. Give a name and leave others to default

4. Now click on the Create button at the bottom. 

5. Now click on container name 

6. Click on Upload button to upload a file

7. Now you need to get the Access Keys

1. Under Settings section, click on Access Keys 

2. Now click on Show Keys button

3. Note down the Key (not connection string) under key1

2. Nifi template for downloading the blobs from a container

8. Go to Nifi interface

9. Create a ListAzureBlobStorage processor.

10. Configure with following properties

1. Container Name: Get the name you have created in previous step

2. Storage Credentials:

1. Create a new Controller Services “AzureStorageCredentialsControllerService”

2. Properties are: Storage Account name (same as you have created before), Storage Account Key (same as you have noted down in previous step)

3. Now Enable the Controller Service 

11. Create a FetchAzureBlobStorage processor with the same properties of  ListAzureBlobStorage processor. 

12. Create a PutFile processor, with the following properties:

1. Directory: /tmp/azure/success (you can change this according to your requirement)

2. Leave rest properties to default

13. Create a PutFile processor, with the following properties. This is just to track if anything goes wrong:

1. Directory: /tmp/azure/fail (you can change this according to your requirement)

2. Leave rest properties to default

14. Now the Nifi template should look like below:

3. Nifi template to UPLOAD files to a container

15. Get the following processors and connect them.

16. Properties of GetFile:

17. Properties of PutAzureBlobStorage processor:

4. Nifi template to delete files/blob from a container

18. Create the following nifi processor:

19. Properties of ListAzureBlobStorage processor:

20. Properties of DeleteAzureBlobStorageprocessor:

Here I am deleting only one blob(i.e. Image 2.png) . But you can leave the Blob properties to its default value to delete all the blobs

Securing Nifi with Google's OAuth 2.0 provider

This post is about securing the NiFi and user will be authorized by Google using OAuth 2.0 login provider.

So the scenario is like this:

• I have my own PC.

• I have Google account with Gmail ID reachchinu(\at)gmail(\dot)com
  

• OpenStack Cloud is provided by the  university where I am working.
  

• I have an instance running in the Openstack cloud.

• IP of that instance is 172.17.66.101.
• NiFi is running here

Current Setup:

 Now I am accessing the Nifi instance with out any security. Anyone with the IP and port with in the the University network can access the Nifi's UI.

What I need:

 NiFi should ask some kind of login info before allowing anyone to access the UI.

What can be done:

Enable the security feature of Nifi. After this, every time you are trying to access Nifi's UI, you need to make yourself authenticated with your Google's OAuth 2.0 provider. 

Pre-requisite:

I have access to that Nifi instance over SSH.

Nifi is installed on /usr/local/bin directories

Steps: 

1. Install and configure Nfi : 

1. Follow URL: https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html 

2. Nifi is installed on /usr/local/bin

3. Nifi version: 1.11.4

4. Stop Nifi service if it is running

cd /usr/local/bin

./nifi-1.11.4/bin/nifi.sh stop

2. Download Nifi toolkit to /usr/local/bin: https://archive.apache.org/dist/nifi/1.11.4/nifi-toolkit-1.11.4-bin.tar.gz 

3. Unzip the downloaded toolkit in the same location

4. Make sure current directory is /usr/local/bin/

5. Execute following command to generate signed certificate for localhost

./nifi-toolkit-1.11.4/bin/tls-toolkit.sh standalone -n "localhost"

6. Copy the new content with following content

cp -rv ./localhost/* /usr/local/bin/nifi-1.11.4/conf/.

7. Open the nifi.properties file present in the /usr/local/bin/nifi-1.11.4/conf directory:

        sudo vim /usr/local/bin/nifi-1.11.4/conf/nifi.properties

8. Edit following line (remove localhost)

nifi.web.https.host=<give here IP>.xip.io

9. Open authorizer.xml file

sudo vim /usr/local/bin/nifi-1.11.4/conf/authorizers.xml

10. Edit following line in <userGroupProvider> section

<property name="Initial User Identity 1">YOUR_ACCOUNT@gmail.com</property>

11. Edit following line in <accessPolicyProvider> section

<property name="Initial Admin Identity">YOUR_ACCOUNT@gmail.com</property>

12. At the end authorizer.xml file should look like below:

<userGroupProvider>         <identifier>file-user-group-provider</identifier>         <class>org.apache.nifi.authorization.FileUserGroupProvider</class>         <property name="Users File">./conf/users.xml</property>         <property name="Legacy Authorized Users File"></property>        <property name="Initial User Identity 1">reachinu@gmail.com</property> </userGroupProvider> <accessPolicyProvider>         <identifier>file-access-policy-provider</identifier>         <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>         <property name="User Group Provider">file-user-group-provider</property>         <property name="Authorizations File">./conf/authorizations.xml</property>         <property name="Initial Admin Identity">reachinu@gmail.com</property>         <property name="Legacy Authorized Users File"></property>         <property name="Node Identity 1"></property>         <property name="Node Group"></property> </accessPolicyProvider>

13. Before proceeding next, have the following information:

1. IP address of the Nifi instance

2. Port number 

         You can get this from /usr/local/bin/nifi-1.11.4/conf/nifi.properties file

The line looks like nifi.web.https.port=8443

14. [src2] Login to the Google Developers Console with your Google account:
    https://console.developers.google.com/apis/credentials 

15. Select project if you want to change as shown in below figure:

16. Click on “CREATE CREDENTIALS” as shown in above figure.

17.  Now select “OAuth client ID”:

18. Fill the next page according to the figure below:
    Application type: 

Name: 

URIs: Change IP address and port number as per your case 

19. Now click on “CREATE” 

20. In the next dialog box, you will see the Client ID and Client secret.

21. 

22. Note down following thing :

1. Your Email add, which is used in the project

2. Client ID

3. Client Secrete 

4. And following url:

https://accounts.google.com/.well-known/openid-configuration 

23. Open the nifi.properties file present in the /usr/local/bin/nifi-1.11.4/conf directory:

nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration nifi.security.user.oidc.connect.timeout=5 secs nifi.security.user.oidc.read.timeout=5 secs nifi.security.user.oidc.client.id=<YOUR-CLIENT-ID> nifi.security.user.oidc.client.secret=<YOUR-CLIENT-SECRET> nifi.security.user.oidc.preferred.jwsalgorithm=

24. At this point, it is expected that the Nifi is stopped

25. Now start the Nifi 

./nifi-1.11.4/bin/nifi.sh start

26. Now go the browser (Here I am using Mozilla firefox) and enter 

https://<ip address >.xip.io:<port number> 

E.g. https://172.17.66.105.xip.io:8443 

 

Click on “Advanced...”

Click on “Accept the Risk and Continue”.

Enter your email address and other details. 

In my case I entered reachinu@gmail.com

 

 

Possible Errors:

1. If you put wrong email address, you will be prompted following message:

2. In your browser, if you ignore .xip.io part, you will be prompted following message:

3. If you try to access nifi with http instead of https, you will get some random character or something like following: 

References

[src1]: https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html 

[src2]: https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect 

 

Note: This work is partially funded by the European Union’s Horizon 2020 research and innovation project RADON (825040).