Showing posts with label Cloud computing. Show all posts
Showing posts with label Cloud computing. Show all posts

Monday, September 20, 2021

Managing Kubernetes clusters using Rancher [20Sept2021]


 
Here is the Environment for me. You may go with single worker node 

Created Tuesday 07 September 2021

Rancher node: Centos 7 (better to keep this separate from cluster nodes), flavor: m3.xsmall
Master Node: Centos 7, m3.xsmall
Worker Node-1: Centos 7, m3.xsmall
Worker Node-2: Centos 7, m3.xsmall

Both Master and Work Node have allow-all security group attached
Minimal ports required are 80/TCP, 443/TCP

[src: https://rancher.com/docs/rancher/v2.x/en/quick-start-guide/deployment/quickstart-manual-setup/

Common installation

  1. Install Docker

  • Create the daemon file manually. This is also because of some conflict in University's Openstack Environment.

        sudo mkdir -p /etc/docker
        sudo tee /etc/docker/daemon.json <<EOF
	{
	  "exec-opts": ["native.cgroupdriver=systemd"],
	  "log-driver": "json-file",
	  "log-opts": {
	    "max-size": "100m"
	  },
	  "storage-driver": "overlay2",
	  "storage-opts": [
	    "overlay2.override_kernel_check=true"
	  ],

"default-address-pools": [{"base":"172.80.0.0/16","size":24}]

	}
	EOF

  • Now run this command. It will add the official Docker repository, download the latest version of Docker, and install it:
curl -fsSL https://get.docker.com/ | sh
  • After installation has completed, start the Docker daemon:
sudo systemctl start docker
ERROR if docker is unable to start, try to check the /etc/docker/daemon.json file
  • Verify that it’s running:
sudo systemctl status docker
  • enable docker to start on boot:
sudo systemctl enable docker
  • To run docker commands with non-root privileges

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Rancher node

  1. Enter the following command to run rancher container
sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher
  1. Go to the web browser and hit https://<SERVER_IP_Rancher_Node>
  2. Follow the wizard for initial setup
Once it is done, you should be able to see the dashboard with local as the cluster name

Create a new cluster

src: https://rancher.com/docs/rancher/v2.5/en/quick-start-guide/deployment/quickstart-manual-setup/

  1. Go to the browser and access Rancher Dashboard
  2. From the dashboard, click on Create
  3. Click on Custom
  4. Give Cluster Name as cluster-1 and skipp other infos
  5. Click on Next
  6. Select etcd , Control Plane , Worker
  7. Select the Registration command
    1. The registration command should look like:
sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.6.0 --server https://172.17.90.86 --token bflxwjlsvsrtbnvp8nj5xq82xx2sr5npjsxm82mbtdcfkc9g65x5d8 --ca-checksum f9bc8c23fff67155023fde69026ec83a77f632657f1049fd6ca9ae5732cf59d3 --etcd --controlplane --worker
  1. Go to master node terminal and execute the registration command

At this point you need to wait for few minutes...

  1. After this, you will see in the browser that 1 New node has registered.
  2. Now click on Done
  3. Repeat Step 6 to get the registration command for worker node. For worker node you just need to select Worker

Deleting a node

[src: https://rancher.com/docs/rancher/v2.5/en/cluster-admin/cleaning-cluster-nodes/]
Deleting Docker Containers, Images, and Volumes

Based on what role you assigned to the node, there are Kubernetes components in containers, containers belonging to overlay networking, DNS, ingress controller and Rancher agent. (and pods you created that have been scheduled to this node)

To clean all Docker containers, images and volumes:

docker rm -f $(docker ps -qa)
docker rmi -f $(docker images -q)
docker volume rm $(docker volume ls -q)

Clean the related directories

sudo rm -rf /etc/ceph \
	   /etc/cni \
	   /etc/kubernetes \
	   /opt/cni \
	   /opt/rke \
	   /run/secrets/kubernetes.io \
	   /run/calico \
	   /run/flannel \
	   /var/lib/calico \
	   /var/lib/etcd \
	   /var/lib/cni \
	   /var/lib/kubelet \
	   /var/lib/rancher/rke/log \
	   /var/log/containers \
	   /var/log/kube-audit \
	   /var/log/pods \
	   /var/run/calico

It is now good to reboot the VM with following command:

sudo reboot



-That's all

Saturday, April 3, 2021

Cloud Cost Comparison : 1vm/mo

 In this post, we will explore and compare the cost of hosting a small VM for 1 month in different clouds. 

1. Serverspace

https://serverspace.us/services/cloud-servers/
Pricing Ref:         https://serverspace.us/pricing/
Configuration: Ubuntu 18.04, 1GB, 1core, 50Mbps, 25GB SSD
Cost/mo:         USD 4.55
Cost/hr:              USD 0.006


2. Kamatera Cloud

https://www.kamatera.com/express/compute/
Configuration: Ubuntu 18.04, 1GB, 1core, 50Mbps, 20GB SSD
Cost/mo:         USD 4.00
Cost/hr:                 USD 0.005
Pricing Ref:         https://www.kamatera.com/express/compute/?tcampaign=35187_368194&bta=35187&nci=5344


3. Linode

https://www.linode.com
Configuration: 1GB, 1core, 25GB SSD, 1TB, 40GB NW In, 1000Mbps NW Out
Cost/mo:         USD 5.00
Cost/mo:         USD 0.0075
Pricing Ref:         https://www.linode.com/pricing/


4. ScalaHosting

https://www.scalahosting.com/cloud-servers
Configuration: Ubuntu 20.04, 2GB, 1core, 50GB SSD, 3000GB bandwidth
Cost/mo:         USD 10.00
Cost/hr:                 USD 0.015
Pricing Ref:         https://my.scalahosting.com/order.php?a=configure&i=0


5. Cloudways

https://www.cloudways.com/
Using there interface you only can create VMs in other cloud provideres, such as DigitalOcean, Linode, VULTR, AWS, Google Cloud
Pricing Ref: https://www.cloudways.com/en/pricing.php


6. LiquidWeb

https://www.liquidweb.com/products/cloud-dedicated/
Configuration: Linux, 2GB, 2 vCPU, 40GB SSD, 10TB bandwidth
Cost/mo:             USD 15 (if you go for 24months plan)
Other Addon:        InterWorx, Plesk Web Pro, or cPanel Admin
Ref:                         https://www.liquidweb.com/products/vps/


7. SiteGround

https://www.siteground.com/cloud-hosting.htm
Configuration: 8GB, 4cores, 40GB SSD, 5TB data transfer
Cost/mo:         USD 80.00
Pricing Ref:         https://www.siteground.com/cloud-hosting.htm?afcode=b0dbbe40d137d9fab687993c2f1ce093&campaign=Cloud+Hosting


8. DigtialOcean

https://www.digitalocean.com/
Configuration: 1GB, 1CPU, 25GB SSD, 1TB data transfer
Cost/mo:         USD 5.00
Cost/hr:                 USD 0.00744
Pricing Ref:         https://www.digitalocean.com/pricing/


9. Vultr

https://www.vultr.com/
Configuration: 512MB, 1CPU, 10GB SSD, 0.5TB data transfer
Cost/mo:         USD 3.5 (can be 2.5USD if you choose IPv6 version only )
Cost/hr:                 USD 0.005
Pricing Ref:         https://www.vultr.com/products/cloud-compute/#pricing


10. Amazon Web Services

https://aws.amazon.com/
Configuration: 1GB, 2vCPU, 10GB HDD, 0.5TB data transfer
Cost/mo:         USD 4.02 
Cost/hr:                 USD 0.005
Pricing Ref:         https://calculator.aws/#/createCalculator/EC2  



Wednesday, December 9, 2020

Integrating Apache Nifi with Azure Storage

 In this post we will see how Apache Nifi can be used to handle the blobs/files in Azure Blob storage.

We will go through:

  • Creating a Storage account in Microsoft Azure
  • Creating Nifi template that download the files/blobs from Azure blob storage
  • Creating a Nifi template that upload files to the blob storage
  • Creating a Nifi template that delete blobs/files from the blob storage

Prerequisites:

It is expected that, you have fundamental knowledge on 

Steps:


1. Create the Storage Account

  1. Go to Azure home portal: https://portal.azure.com/?quickstart=True#home 

  2. Click on Storage Account :

  1. Click on Add Button, if you don’t have a Storage account. Below figures are for your reference:










  1. Now go to the Storage Account portal: https://portal.azure.com/?quickstart=True#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts 

  2. Click on the storage account name

  3.  Under “Blob service” section

    1. Click on Containers.

    2. Click + Container button to create a new container

    3. Give a name and leave others to default

    4. Now click on the Create button at the bottom. 

    5. Now click on container name 

    6. Click on Upload button to upload a file

  4. Now you need to get the Access Keys

    1. Under Settings section, click on Access Keys 

    2. Now click on Show Keys button

    3. Note down the Key (not connection string) under key1

2. Nifi template for downloading the blobs from a container

  1. Go to Nifi interface

  2. Create a ListAzureBlobStorage processor.

  3. Configure with following properties

    1. Container Name: Get the name you have created in previous step

    2. Storage Credentials:

      1. Create a new Controller Services “AzureStorageCredentialsControllerService”

      2. Properties are: Storage Account name (same as you have created before), Storage Account Key (same as you have noted down in previous step)

      3. Now Enable the Controller Service 

  4. Create a FetchAzureBlobStorage processor with the same properties of  ListAzureBlobStorage processor. 

  5. Create a PutFile processor, with the following properties:

    1. Directory: /tmp/azure/success (you can change this according to your requirement)

    2. Leave rest properties to default

  6. Create a PutFile processor, with the following properties. This is just to track if anything goes wrong:

    1. Directory: /tmp/azure/fail (you can change this according to your requirement)

    2. Leave rest properties to default

  7. Now the Nifi template should look like below:

3. Nifi template to UPLOAD files to a container

  1. Get the following processors and connect them.

  1. Properties of GetFile:

  1. Properties of PutAzureBlobStorage processor:


4. Nifi template to delete files/blob from a container

  1. Create the following nifi processor:

  1. Properties of ListAzureBlobStorage processor:

  1. Properties of DeleteAzureBlobStorageprocessor:

Here I am deleting only one blob(i.e. Image 2.png) . But you can leave the Blob properties to its default value to delete all the blobs




Friday, December 4, 2020

Securing Nifi with Google's OAuth 2.0 provider

This post is about securing the NiFi and user will be authorized by Google using OAuth 2.0 login provider.


So the scenario is like this:

  • I have my own PC.
  • I have Google account with Gmail ID reachchinu(\at)gmail(\dot)com
  • OpenStack Cloud is provided by the  university where I am working.
  • I have an instance running in the Openstack cloud.
  • IP of that instance is 172.17.66.101.
  • NiFi is running here

Current Setup:

 Now I am accessing the Nifi instance with out any security. Anyone with the IP and port with in the the University network can access the Nifi's UI.

What I need:

 NiFi should ask some kind of login info before allowing anyone to access the UI.

What can be done:

Enable the security feature of Nifi. After this, every time you are trying to access Nifi's UI, you need to make yourself authenticated with your Google's OAuth 2.0 provider. 

Pre-requisite:

I have access to that Nifi instance over SSH.

Nifi is installed on /usr/local/bin directories

Steps: 

  1. Install and configure Nfi : 

    1. Follow URL: https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html 

    2. Nifi is installed on /usr/local/bin

    3. Nifi version: 1.11.4

    4. Stop Nifi service if it is running

cd /usr/local/bin

./nifi-1.11.4/bin/nifi.sh stop


  1. Download Nifi toolkit to /usr/local/bin: https://archive.apache.org/dist/nifi/1.11.4/nifi-toolkit-1.11.4-bin.tar.gz 

  2. Unzip the downloaded toolkit in the same location

  3. Make sure current directory is /usr/local/bin/

  4. Execute following command to generate signed certificate for localhost

./nifi-toolkit-1.11.4/bin/tls-toolkit.sh standalone -n "localhost"

  1. Copy the new content with following content

cp -rv ./localhost/* /usr/local/bin/nifi-1.11.4/conf/.

  1. Open the nifi.properties file present in the /usr/local/bin/nifi-1.11.4/conf directory:

        sudo vim /usr/local/bin/nifi-1.11.4/conf/nifi.properties

  1. Edit following line (remove localhost)

nifi.web.https.host=<give here IP>.xip.io

  1. Open authorizer.xml file

sudo vim /usr/local/bin/nifi-1.11.4/conf/authorizers.xml

  1. Edit following line in <userGroupProvider> section

<property name="Initial User Identity 1">YOUR_ACCOUNT@gmail.com</property>

  1. Edit following line in <accessPolicyProvider> section

<property name="Initial Admin Identity">YOUR_ACCOUNT@gmail.com</property>

  1. At the end authorizer.xml file should look like below:

<userGroupProvider>

        <identifier>file-user-group-provider</identifier>

        <class>org.apache.nifi.authorization.FileUserGroupProvider</class>

        <property name="Users File">./conf/users.xml</property>

        <property name="Legacy Authorized Users File"></property>

       <property name="Initial User Identity 1">reachinu@gmail.com</property>

</userGroupProvider>


<accessPolicyProvider>

        <identifier>file-access-policy-provider</identifier>

        <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>

        <property name="User Group Provider">file-user-group-provider</property>

        <property name="Authorizations File">./conf/authorizations.xml</property>

        <property name="Initial Admin Identity">reachinu@gmail.com</property>

        <property name="Legacy Authorized Users File"></property>

        <property name="Node Identity 1"></property>

        <property name="Node Group"></property>

</accessPolicyProvider>



  1. Before proceeding next, have the following information:

    1. IP address of the Nifi instance

    2. Port number 

         You can get this from /usr/local/bin/nifi-1.11.4/conf/nifi.properties file

The line looks like nifi.web.https.port=8443

  1. [src2] Login to the Google Developers Console with your Google account:
    https://console.developers.google.com/apis/credentials 

  2. Select project if you want to change as shown in below figure:

  1. Click on “CREATE CREDENTIALS” as shown in above figure.

  2.  Now select “OAuth client ID”:

  1. Fill the next page according to the figure below:
    Application type: 

Name: 

URIs: Change IP address and port number as per your case 

  1. Now click on “CREATE” 

  2. In the next dialog box, you will see the Client ID and Client secret.

  3. Note down following thing :

    1. Your Email add, which is used in the project

    2. Client ID

    3. Client Secrete 

    4. And following url:

https://accounts.google.com/.well-known/openid-configuration 

  1. Open the nifi.properties file present in the /usr/local/bin/nifi-1.11.4/conf directory:


nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration

nifi.security.user.oidc.connect.timeout=5 secs

nifi.security.user.oidc.read.timeout=5 secs

nifi.security.user.oidc.client.id=<YOUR-CLIENT-ID>

nifi.security.user.oidc.client.secret=<YOUR-CLIENT-SECRET>

nifi.security.user.oidc.preferred.jwsalgorithm=


  1. At this point, it is expected that the Nifi is stopped

  2. Now start the Nifi 

./nifi-1.11.4/bin/nifi.sh start

  1. Now go the browser (Here I am using Mozilla firefox) and enter 

https://<ip address >.xip.io:<port number> 

E.g. https://172.17.66.105.xip.io:8443 

 

Click on “Advanced...”

Click on “Accept the Risk and Continue”.

Enter your email address and other details. 

In my case I entered reachinu@gmail.com


 
 

Possible Errors:


  1. If you put wrong email address, you will be prompted following message:


  1. In your browser, if you ignore .xip.io part, you will be prompted following message:


  1. If you try to access nifi with http instead of https, you will get some random character or something like following: 



References

[src1]: https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html 

[src2]: https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect 

 

Note: This work is partially funded by the European Union’s Horizon 2020 research and innovation project RADON (825040).