Azure Databricks

 It is an Apache Spark based analytics paltform optimized for Microsoft axzure

key features

1. Spark SQL lirabry
2. Streaming services for OIT et
3. MLIB machine leaning libs
4. Graph Computation

Spark core API supports

1. R
2. SQL
3. Python
4. Scala
5. Java

Azure Databricks as a pltform contains.

1. Datbricks workspace
2. Datbrcisk workflows
3. Databricks Runtime
4. Databricks I/O
5. Datbricks Serverless
6. Dataricks Enterprise Security (DBES)

Storage solutions supported

1. Blob storage
2. Data lake
3. SQL DW
4. Apache kafka
5. Hadoop

Applications Supported

1. ML
2. Streaming
3. Data 
4. Power BI
5. Others

Users

• Data Scientist
• Data engineers
• Analysts
• othes

Azure Data Factory (Rest End Point one side and Storage in other end point)

1. Need to have REST API2. Create Data Lake Storage3. Create Azure Data Factory4. Create App registration to access data from Data Lake

Get the URL 

Test URL like Postman or something.

Create data lake storage

Create app registration

Create two linked services ( one for rest end point and other for data lake storage)

RStudio Setup

 

Windows

• Download R for Windows here
• Run the .exe file that was downloaded in the step above.
• Go to the RStudio Download page
• Under Installers select RStudio current version ## - Windows XP/Vista/7/8/10.
• Double click the file to install it.

Once R and RStudio are installed, open RStudio to make sure that you don’t get any error messages.

AI - Important Terminology

SOLID Principles

S - Single Responsibility Principle

O - Open/Closed Principle

L - Liskov Substitution Principle

I - Interface Segregation Principle

D - Dependency Inversion  Principle 

HadoopSetup

http://www.hadoopwizard.com/install-hadoop-on-windows-in-3-easy-steps-for-hortonworks-sandbox-tutorial/

https://hortonworks.com/downloads/#sandbox

https://www.youtube.com/watch?v=7sxqHgBdxB8

https://www.youtube.com/watch?v=xWgdny19yQ4

https://www.virtualbox.org/wiki/Downloads


option-1
------------
https://www.virtualbox.org/wiki/Downloads

Blockchain - Steps to Setup Environment

1) install tuffleframeword.com/ganache

2) start it..

3) check npm version

4) install python 3.7.4 for windows

5) npm -g install truffle@5.0.2

6) install metamask google echrome extension --> allows to connect to BC personalle account


7) truffle version

8) truffle init

9) packahe.jso

10) Ganache,, from truffle website

11) Metamask extension from google

12) syntax highlighting for solidity.. SUbmile text

------------------------------------------
1) create the app folder    --> mkdir election

2) unbox pet-shop project from truffle --> truffle unbox pet-shop

Docker Commands

ddocker display version

ddocker login

ddocker pull name-of-the-image

ddocker pull webdevops/php-apache

ddocker search laravel

ddocker images

ddocker rmi IMAGE-ID

ddocker run IMAGE-ID

ddocker ps

ddocker stop CONTAINER-ID

ddocker ps -a

ddocker rm [ID]