Friday, December 4, 2020

Securing Nifi with Google's OAuth 2.0 provider

This post is about securing the NiFi and user will be authorized by Google using OAuth 2.0 login provider.

So the scenario is like this:

I have my own PC.

I have Google account with Gmail ID reachchinu(\at)gmail(\dot)com

OpenStack Cloud is provided by the university where I am working.

I have an instance running in the Openstack cloud.

IP of that instance is 172.17.66.101.
NiFi is running here

Current Setup:

Now I am accessing the Nifi instance with out any security. Anyone with the IP and port with in the the University network can access the Nifi's UI.

What I need:

NiFi should ask some kind of login info before allowing anyone to access the UI.

What can be done:

Enable the security feature of Nifi. After this, every time you are trying to access Nifi's UI, you need to make yourself authenticated with your Google's OAuth 2.0 provider.

Pre-requisite:

I have access to that Nifi instance over SSH.

Nifi is installed on /usr/local/bin directories

Steps:

Install and configure Nfi :

Follow URL: https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
Nifi is installed on /usr/local/bin
Nifi version: 1.11.4
Stop Nifi service if it is running

cd /usr/local/bin

./nifi-1.11.4/bin/nifi.sh stop

Download Nifi toolkit to /usr/local/bin: https://archive.apache.org/dist/nifi/1.11.4/nifi-toolkit-1.11.4-bin.tar.gz
Unzip the downloaded toolkit in the same location
Make sure current directory is /usr/local/bin/
Execute following command to generate signed certificate for localhost

./nifi-toolkit-1.11.4/bin/tls-toolkit.sh standalone -n "localhost"

Copy the new content with following content

cp -rv ./localhost/* /usr/local/bin/nifi-1.11.4/conf/.

Open the nifi.properties file present in the /usr/local/bin/nifi-1.11.4/conf directory:

sudo vim /usr/local/bin/nifi-1.11.4/conf/nifi.properties

Edit following line (remove localhost)

nifi.web.https.host=<give here IP>.xip.io

Open authorizer.xml file

sudo vim /usr/local/bin/nifi-1.11.4/conf/authorizers.xml

Edit following line in <userGroupProvider> section

<property name="Initial User Identity 1">YOUR_ACCOUNT@gmail.com</property>

Edit following line in <accessPolicyProvider> section

<property name="Initial Admin Identity">YOUR_ACCOUNT@gmail.com</property>

At the end authorizer.xml file should look like below:

<identifier>file-user-group-provider</identifier>

<class>org.apache.nifi.authorization.FileUserGroupProvider</class>

<property name="Users File">./conf/users.xml</property>

<property name="Initial User Identity 1">reachinu@gmail.com</property>

</userGroupProvider>

<identifier>file-access-policy-provider</identifier>

<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>

<property name="User Group Provider">file-user-group-provider</property>

<property name="Authorizations File">./conf/authorizations.xml</property>

<property name="Initial Admin Identity">reachinu@gmail.com</property>

</accessPolicyProvider>

Before proceeding next, have the following information:

IP address of the Nifi instance
Port number

You can get this from /usr/local/bin/nifi-1.11.4/conf/nifi.properties file

The line looks like nifi.web.https.port=8443

[src2] Login to the Google Developers Console with your Google account:
https://console.developers.google.com/apis/credentials
Select project if you want to change as shown in below figure:

Click on “CREATE CREDENTIALS” as shown in above figure.
Now select “OAuth client ID”:

Fill the next page according to the figure below:
Application type:

Name:

URIs: Change IP address and port number as per your case

Now click on “CREATE”
In the next dialog box, you will see the Client ID and Client secret.
Note down following thing :

Your Email add, which is used in the project
Client ID
Client Secrete
And following url:

https://accounts.google.com/.well-known/openid-configuration

Open the nifi.properties file present in the /usr/local/bin/nifi-1.11.4/conf directory:

nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration

nifi.security.user.oidc.connect.timeout=5 secs

nifi.security.user.oidc.read.timeout=5 secs

nifi.security.user.oidc.client.id=<YOUR-CLIENT-ID>

nifi.security.user.oidc.client.secret=<YOUR-CLIENT-SECRET>

nifi.security.user.oidc.preferred.jwsalgorithm=

At this point, it is expected that the Nifi is stopped
Now start the Nifi

./nifi-1.11.4/bin/nifi.sh start

Now go the browser (Here I am using Mozilla firefox) and enter

https://<ip address >.xip.io:<port number>

E.g. https://172.17.66.105.xip.io:8443

Click on “Advanced...”

Click on “Accept the Risk and Continue”.

Enter your email address and other details.

In my case I entered reachinu@gmail.com

Possible Errors:

If you put wrong email address, you will be prompted following message:

In your browser, if you ignore .xip.io part, you will be prompted following message:

If you try to access nifi with http instead of https, you will get some random character or something like following:

References

[src1]: https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html

[src2]: https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect

Note: This work is partially funded by the European Union’s Horizon 2020 research and innovation project RADON (825040).

Saturday, May 4, 2019

Ansible: Installation and Configuration

Ansible is an open-source software provisioning, configuration management, and application deployment tool. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows. Before proceeding for the installation, it is necessary to get yourself comfortable with basic knowledge.

Ansible Wikipedia: https://en.wikipedia.org/wiki/Ansible_(software)
Official Documentation: https://docs.ansible.com/

1. Ansible installation

1.1. Through pip

The easy way to install ansible is through pip, which is a python package manager.

Prerequisite:

> For before installing ansible, make sure that python3 is already installed.

> Once Python3 is installed go ahead to install pip

Sudo easy-install pip3

> Now install ansible through pip3

pip3 install ansible

1.2. Through apt (ubuntu)

$ sudo apt-get update

$ sudo apt-get install software-properties-common

$ sudo apt-add-repository --yes --update ppa:ansible/ansible

$ sudo apt-get install ansible

[src]: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#latest-releases-via-pip

2. Basic Configuration:

If ansible is installed manually through pip, the ansible_host inventory file needs to be created manually. Following this you also need to check the reachability of remote servers.

2.1. Adding host file if NOT present

If the ansible is installed using pip3, you will not find any host inventory in the directory /etc/ansible/hosts

if you execute the command ansible webservers --list-hosts this will give an error something like "unable to parse the inventory file in /etc/ansible/hosts". This means that no directory named ansible in /etc/ directory is present. For this follow steps below:

1: cd /etc/

2: sudo mkdir -p ansible/hosts/

3: vim ansible_host

[localhost]

127.0.0.1

[webservers]

[dbservers]

[myhomepage]

www.chinmayadehury.in

[testservers]

www.google.com

www.chinmayadehury.in

4: now enter the command ansible testservers --list-hosts

This will list the webservers that are listed in the group called [testservers]

2.2. Connecting to remote host using ssh

1: open the ansible_host inventory file

vim /etc/ansible/hosts/ansible_host

2: add the following host group or host

[myhomepage]

160.153.137.40 ansible_port=<sshport#> ansible_user=username1 ansible_ssh_pass=user1password

It is assumed that you are able to login to that remote using ssh and using above credentials.

3: now save the ansible_host file

4: enter following command

ansible myhomepage -m ping

2.3. Disabling ssh key host checking

Open the config file in /etc/ansible/ansible.config

Find for #host_key_checking = false. Uncomment the line to make it enable.
[src]: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#latest-releases-via-pip

-Thats all for now.

Tuesday, April 16, 2019

Ansible Playbook: Generate multiple files at specific time interval

In the previous post, we have seen how to generate multiple files using with_sequence and loop features.

Here, we will introduce the delay functionality in the loop feature. This will give a pause after every iteration. So, each file will be created at specific time interval. Let's say, we will create 20 files at 10sec time interval.
Download the yml file [generateFiles_pattern_withDelay.yml]

--- 
- hosts: localhost
  gather_facts: no
  become: no
  tasks:
    - name: create 20 empty file in each 10 sec
      file:
        path: "/tmp/playing_opera/loopdemo/file{{ item }}"
        state: touch
      with_sequence: start=1 end=20
      loop_control:
        pause: 10
...

Next: ..?? Not yet decided :)

Ansible Playbook: Generating multiple files with loop and with_sequence

Let's write a playbook which would create multiple empty files. All the filenames must follow a pattern. Lets say file1, file2, file3, ...

For this we will use with_sequence and loop features. with_sequence would allow us to generate a list of numbers and loop feature would allow us to iterate of the lists.

Generate empty file from a list

Job is to create a list of filenames and use that list to create multiple empty file.
Download yml file [generateFiles1.yml]

--- 
- hosts: localhost
  gather_facts: no
  become: no
  tasks:
    - name: create the location/directory
      file:
        path: /tmp/playing_opera/loopdemo
        state: directory

    - name: generate empty file from a list of file names
      file:
        path: "/tmp/playing_opera/loopdemo/{{ item }}"
        state: touch
      with_items:
        - fruitsList
        - censusReport
        - countryList
…

The first task would first make sure that the directory is present. If not this would create one.
The second task is to create three files with name "fruitsList", "censusReport", and "countryList".

The above playbook can also written in following way.
Download yml file [generateFiles2.yml]

--- 
- hosts: localhost
  gather_facts: no
  become: no
  tasks:
    - name: create the location/directory
      file:
        path: /tmp/playing_opera/loopdemo
        state: directory

    - name: generate empty file from a list of file names
      file:
        path: "/tmp/playing_opera/loopdemo/{{ item }}"
        state: touch
      loop: [ fruitsList, censusReport, countryList ]
…

Generate empty files with pattern

It is sometimes necessary to create a list of files following some patterns. For example file1, file2, file3, ...
This can be achieved using with_sequence feature of ansible.
Following is an example.

Download yml file [generateFiles_pattern.yml]

--- 
- hosts: localhost
  gather_facts: no
  become: no
  tasks:
    - name: generate 10 empty files
      file:
        path: "/tmp/playing_opera/loopdemo/file{{ item }}"
        state: touch
      with_sequence: start=1 end=10
…

This playbook would create 10 files with the names file1, file2, ..., file10.
Explorer the official documentation for more info.

Next: Generate file at specific time interval

Manually adding ansible_host inventory file, if NOT present.

In computing, Ansible is an open-source software provisioning, configuration management, and application deployment tool. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows.

If you are installing ansible using pip or any other manual process, first check if "ansible" directory is present inside "/etc"

if not, here is the steps you can follow to add ansible_host inventory file. This inventory file mainly keeps the inventory of detail information regarding all the host machines.

Steps:

1> Open Terminal in the linux system and enter following commands
2> cd /etc
3> mkdir -p ansible/host/
4> cd ansible/host/
5> vim ansible_host
6> Add host details in following format
[host group name]
<host IP> [ansible_port=<port number>] [ansible_user=<username>] [ansible_ssh_pass=<password>]
e.g.
[localvms]
172.17.12.65 ansible_user=guestuser1 ansible_ssh_pass=123
172.17.12.80 ansible_port=808 ansible_user=guestuser1 ansible_ssh_pass=123

Click here for more information from official source.

Ansible Playbook: ping remote server(s)

ping command is basically used to test the reachability of a server. Ansible allows you to ping multiple servers or a group of servers at once. It is assumed that, Ansible tool is already installed in your linux system.

Steps:
1> Create empty yml file with the name pingserver.yml

vim pingserver.yml

2> Add following contents:

---
- hosts: myservers
  tasks:
   - name: check the ping
     ping:
...

3> Save and execute the pingserver.yml file using followng command

ansible-playbook pingserver.yml

make sure following content is present in ansible_host inventory file

[myservers]

192.168.15.34 ansible_user=chinmaya ansible_ssh_pass=mypassword

www.chinmayadehury.in ansible_user=chinmaya ansible_ssh_pass=mypassword

192.168.15.37 ansible_user=chinmaya ansible_ssh_pass=mypassword

[Alternative]
In this method no playbook file is needed to create.
Pass the following command to the terminal

ansible myservers -m ping

If ansible_host file is not present, click here for the steps to add ansible_host file.

-Thanks

Computer Inquisitive

Pages