Thursday, August 6, 2020

Azure Databricks

 It is an Apache Spark based analytics paltform optimized for Microsoft axzure



key features

  1. Spark SQL lirabry
  2. Streaming services for OIT et
  3. MLIB machine leaning libs
  4. Graph Computation


Spark core API supports

  1. R
  2. SQL
  3. Python
  4. Scala
  5. Java

Azure Databricks as a pltform contains.

  1. Datbricks workspace
  2. Datbrcisk workflows
  3. Databricks Runtime
  4. Databricks I/O
  5. Datbricks Serverless
  6. Dataricks Enterprise Security (DBES)
Storage solutions supported
  1. Blob storage
  2. Data lake
  3. SQL DW
  4. Apache kafka
  5. Hadoop
Applications Supported
  1. ML
  2. Streaming
  3. Data 
  4. Power BI
  5. Others
Users
  • Data Scientist
  • Data engineers
  • Analysts
  • othes



Azure Data Factory (Rest End Point one side and Storage in other end point)

1. Need to have REST API2. Create Data Lake Storage3. Create Azure Data Factory4. Create App registration to access data from Data Lake


Get the URL 
Test URL like Postman or something.

Create data lake storage
Create app registration
Create two linked services ( one for rest end point and other for data lake storage)

RStudio Setup

 

Windows

Once R and RStudio are installed, open RStudio to make sure that you don’t get any error messages.

Saturday, February 22, 2020

AI - Important Terminology



SOLID Principles

S - Single Responsibility Principle

O - Open/Closed Principle

L - Liskov Substitution Principle

I - Interface Segregation Principle

D - Dependency Inversion  Principle 

Friday, February 14, 2020

HadoopSetup

http://www.hadoopwizard.com/install-hadoop-on-windows-in-3-easy-steps-for-hortonworks-sandbox-tutorial/

https://hortonworks.com/downloads/#sandbox

https://www.youtube.com/watch?v=7sxqHgBdxB8

https://www.youtube.com/watch?v=xWgdny19yQ4

https://www.virtualbox.org/wiki/Downloads


option-1
------------
https://www.virtualbox.org/wiki/Downloads

Blockchain - Steps to Setup Environment

1) install tuffleframeword.com/ganache

2) start it..

3) check npm version

4) install python 3.7.4 for windows

5) npm -g install truffle@5.0.2

6) install metamask google echrome extension --> allows to connect to BC personalle account


7) truffle version

8) truffle init

9) packahe.jso

10) Ganache,, from truffle website

11) Metamask extension from google

12) syntax highlighting for solidity.. SUbmile text

------------------------------------------
1) create the app folder    --> mkdir election

2) unbox pet-shop project from truffle --> truffle unbox pet-shop


Docker Commands

ddocker display version
ddocker login
ddocker pull name-of-the-image
ddocker pull webdevops/php-apache
ddocker search laravel
ddocker images
ddocker rmi IMAGE-ID
ddocker run IMAGE-ID
ddocker ps
ddocker stop CONTAINER-ID
ddocker ps -a
ddocker rm [ID]

Thursday, December 19, 2019

12 Factor Guidelines



    The twelve-factor app is a methodology for building software-as-a-service apps that:


    I. Codebase

    One codebase tracked in revision control, many deploys

    II. Dependencies

    Explicitly declare and isolate dependencies

    III. Config

    Store config in the environment

    IV. Backing services

    Treat backing services as attached resources

    V. Build, release, run

    Strictly separate build and run stages

    VI. Processes

    Execute the app as one or more stateless processes

    VII. Port binding

    Export services via port binding

    VIII. Concurrency

    Scale out via the process model

    IX. Disposability

    Maximize robustness with fast startup and graceful shutdown

    X. Dev/prod parity

    Keep development, staging, and production as similar as possible

    XI. Logs

    Treat logs as event streams

    XII. Admin processes

    Run admin/management tasks as one-off processes

    Wednesday, December 18, 2019

    Spring BOOT Annotations and Working Demo

    Spring is a popular Java application framework for creating enterprise applications. Spring Boot is the next step in evolution of Spring framework. It helps create stand-alone, production-grade Spring based applications with minimal effort. It does not use XML configurations anymore and implements the convention over configuration principle.
    Annotation is a form of metadata which provides data about a program that is not part of the program itself. Annotations do not have direct effect on the operation of the code they annotate.

    Spring Boot basic annotations

    In the example application, we have these Spring Boot annotations:
    • @Bean - indicates that a method produces a bean to be managed by Spring.
    • @Service - indicates that an annotated class is a service class.
    • @Repository - indicates that an annotated class is a repository, which is an abstraction of data access and storage.
    • @Configuration - indicates that a class is a configuration class that may contain bean definitions.
    • @Controller - marks the class as web controller, capable of handling the requests.
    • @RequestMapping - maps HTTP request with a path to a controller method.
    • @Autowired - marks a constructor, field, or setter method to be autowired by Spring dependency injection.
    • @SpringBootApplication - enables Spring Boot autoconfiguration and component scanning.
    @Component is a generic stereotype for a Spring managed component. It turns the class into a Spring bean at the auto-scan time. Classes decorated with this annotation are considered as candidates for auto-detection when using annotation-based configuration and classpath scanning. @Repository@Service, and @Controller are specializations of @Component for more specific use cases.
    There are also Hibernate @Entity@Table@Id, and @GeneratedValue annotations in the example.

    Spring Boot basic annotations example

    The following application is a Spring Boot application which returns data from an H2 database using Spring Data JPA. The application uses FreeMarker as a template engine.
    pom.xml
    src
    ├───main
    │   ├───java
    │   │   └───com
    │   │       └───zetcode
    │   │           │   Application.java
    │   │           ├───controller
    │   │           │       MyController.java
    │   │           ├───model
    │   │           │       City.java
    │   │           ├───repository
    │   │           │       CityRepository.java
    │   │           └───service
    │   │                   CityService.java
    │   │                   ICityService.java
    │   └───resources
    │       │   application.yaml
    │       │   import.sql
    │       ├───static
    │       │   └───css
    │       │           style.css
    │       └───templates
    │               index.ftl
    │               showCities.ftl
    └───test
        └───java
    
    This is the project structure.
    pom.xml
    <?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
             http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
    
        <groupId>swam.com</groupId>
        <artifactId>springbootbasicannotations</artifactId>
        <version>1.0-SNAPSHOT</version>
        <packaging>jar</packaging>
    
        <properties>
            <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
            <maven.compiler.source>11</maven.compiler.source>
            <maven.compiler.target>11</maven.compiler.target>
        </properties>
    
        <parent>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-parent</artifactId>
            <version>2.1.1.RELEASE</version>
        </parent>
    
        <dependencies>
    
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-web</artifactId>
            </dependency>
    
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-freemarker</artifactId>
            </dependency>
    
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-data-jpa</artifactId>
            </dependency>
    
            <dependency>
                <groupId>com.h2database</groupId>
                <artifactId>h2</artifactId>
                <scope>runtime</scope>
            </dependency>
    
        </dependencies>
    
        <build>
            <plugins>
                <plugin>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                </plugin>
            </plugins>
        </build>
    
    </project>
    

    This is the Maven build file. It contains dependencies for Freemaker, Spring Data JPA, and H2 database. When Spring Boot finds Freemaker and H2 in the pom.xml, it automatically configures them. We can use them right away.

    resources/application.yml
    server:
      servlet:
        context-path: /myapp
    
    spring:
      main:
        banner-mode: "off"
    datasource:
        platform: h2
    
    logging:
      level:
        org:
          springframework: ERROR
    
    In the application.yml file we write various configuration settings of a Spring Boot application.
    com/zetcode/model/City.java
    package swam.com.model;
    
    import java.util.Objects;
    import javax.persistence.Entity;
    import javax.persistence.GeneratedValue;
    import javax.persistence.GenerationType;
    import javax.persistence.Id;
    import javax.persistence.Table;
    
    @Entity
    @Table(name = "cities")
    public class City {
    
        @Id
        @GeneratedValue(strategy = GenerationType.IDENTITY)
        private Long id;
    
        private String name;
        private int population;
    
        public City() {
        }
    
        public City(String name, int population) {
            this.name = name;
            this.population = population;
        }
    
        public Long getId() {
            return id;
        }
    
        public void setId(Long id) {
            this.id = id;
        }
    
        public String getName() {
            return name;
        }
    
        public void setName(String name) {
            this.name = name;
        }
    
        public int getPopulation() {
            return population;
        }
    
        public void setPopulation(int population) {
            this.population = population;
        }
    
        @Override
        public int hashCode() {
            int hash = 7;
            hash = 79 * hash + Objects.hashCode(this.id);
            hash = 79 * hash + Objects.hashCode(this.name);
            hash = 79 * hash + this.population;
            return hash;
        }
    
        @Override
        public boolean equals(Object obj) {
            if (this == obj) {
                return true;
            }
            if (obj == null) {
                return false;
            }
            if (getClass() != obj.getClass()) {
                return false;
            }
            final City other = (City) obj;
            if (this.population != other.population) {
                return false;
            }
            if (!Objects.equals(this.name, other.name)) {
                return false;
            }
            return Objects.equals(this.id, other.id);
        }
    
        @Override
        public String toString() {
    
            var builder = new StringBuilder();
            builder.append("City{id=").append(id).append(", name=")
                    .append(name).append(", population=")
                    .append(population).append("}");
    
            return builder.toString();
        }
    }
    

    This is the City entity. Each entity must have at least two annotations defined: @Entity and @Id. The @Entity annotation specifies that the class is an entity and is mapped to a database table. The @Table annotation specifies the name of the database table to be used for mapping. The @Id annotation specifies the primary key of an entity and the @GeneratedValue provides for the specification of generation strategies for the values of primary keys.

    resources/import.sql
    INSERT INTO cities(name, population) VALUES('Bratislava', 432000);
    INSERT INTO cities(name, population) VALUES('Budapest', 1759000);
    INSERT INTO cities(name, population) VALUES('Prague', 1280000);
    INSERT INTO cities(name, population) VALUES('Warsaw', 1748000);
    INSERT INTO cities(name, population) VALUES('Los Angeles', 3971000);
    INSERT INTO cities(name, population) VALUES('New York', 8550000);
    INSERT INTO cities(name, population) VALUES('Edinburgh', 464000);
    INSERT INTO cities(name, population) VALUES('Berlin', 3671000);
    
    The schema is automatically created by Hibernate; later, the import.sql file is executed to fill the table with data.
    com/zetcode/repository/CityRepository.java
    package swam.com.repository;
    
    import swam.com.model.City;
    import org.springframework.data.repository.CrudRepository;
    import org.springframework.stereotype.Repository;
    
    @Repository
    public interface CityRepository extends CrudRepository<City, Long> {
    
    }
    
    The @Repository annotation is used to define a repository.
    com/zetcode/service/ICityService.java
    package swam.com.service;
    
    import swam.com.model.City;
    import java.util.List;
    
    public interface ICityService {
    
        List<City> findAll();
    }
    
    ICityService provides a contract method to get all cities.
    swam/com/service/CityService.java
    package swam.com.service;
    
    import swam.com.model.City;
    import swam.com.repository.CityRepository;
    import java.util.List;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.stereotype.Service;
    
    @Service
    public class CityService implements ICityService {
    
        @Autowired
        private CityRepository cityRepository;
    
        @Override
        public List<City> findAll() {
    
            return (List<City>) cityRepository.findAll();
    
        }
    }
    
    The @Service annotation declares CityService to be a service class; a class that provides business services. The @Autowired annotation marks cityRepository field to be injected with CityRepository.
    swam/com/controller/MyController.java
    package swam.com.controller;
    
    import swam.com.service.ICityService;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.stereotype.Controller;
    import org.springframework.ui.Model;
    import org.springframework.web.bind.annotation.RequestMapping;
    import org.springframework.web.servlet.ModelAndView;
    
    import java.util.HashMap;
    import java.util.Map;
    
    @Controller
    public class MyController {
    
        @Autowired
        private ICityService cityService;
    
        @RequestMapping("/")
        public String index(Model model) {
    
            return "index";
        }
    
        @RequestMapping("/cities")
        public ModelAndView showCities() {
    
            var cities = cityService.findAll();
    
            Map<String, Object> params = new HashMap<>();
            params.put("cities", cities);
    
            return new ModelAndView("showCities", params);
        }
    }
    
    The @Controller annotation marks a class as a web controller. The @RequestMapping maps HTTP request with a path to a controller method. In the second case, it maps the /cities URL to the showCities() method.
    resources/templates/index.ftl
    <!DOCTYPE html>
    <html>
        <head>
            <title>Home page</title>
            <meta charset="UTF-8">
            <meta name="viewport" content="width=device-width, initial-scale=1.0">
        </head>
        <body>
            <a href="cities">Show cities</a>
        </body>
    </html>
    
    This is the index.ftl template file. It contains a link to create a request to show all cities.
    resources/templates/showCities.ftl
    <!DOCTYPE html>
    <html>
        <head>
            <title>Cities</title>
            <meta charset="UTF-8">
            <meta name="viewport" content="width=device-width, initial-scale=1.0">
            <link rel="stylesheet" href="css/style.css">
        </head>
        <body>
            <h2>List of cities</h2>
    
            <table>
                <tr>
                    <th>Id</th>
                    <th>Name</th>
                    <th>Population</th>
                </tr>
    
                <#list cities as city>
                    <tr>
                        <td>${city.id}</td>
                        <td>${city.name}</td>
                        <td>${city.population}</td>
                    </tr>
                </#list>
            </table>
        </body>
    </html>
    
    This is the showCities.ftl template file. It uses FreeMarker #list macro to display all city objects.
    resources/static/css/style.css
    h2 {color: blue}
    
    td:nth-child(3) {
        text-align: right;
    }
    
    This is the style.css template file.
    com/zetcode/Application.java
    package swam.com;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    
    @SpringBootApplication
    public class Application {
    
        public static void main(String[] args) {
            SpringApplication.run(Application.class, args);
        }
    }
    


    The @SpringBootApplication enables auto-configuration and component scanning.
    $ mvn spring-boot:run 
    
    We run the application and locate to the localhost:8080/myapp address.
    In this tutorial, we have covered a few basic Spring Boot annotations.