HDP Security Overview

Hortonworks Inc. 2011 2015. All Rights Reserved

HadoopYifeng JiangMarch 10, 2015


(Yifeng Jiang)

Solutions Engineer @ Hortonworks JapanHBase Book Author@uprush


Hadoop REST API


5

YARN Data Lake

2014 65% 2013


Kerberos

?

HDP 2.2

RANGER


DB

DB

D A T A

HADOOP


HDFS

End to End

HiveServer 2 A B C

KDC

Use Hive ST, submit query

Hive gets Namenode (NN) service ticket

6.Hive creates MR/ Tez using NN ST

Ranger

3.Knox gets service ticket (ST) for Hive

4.Knox calls as proxy user

1.Original request w/user id/password

Client gets query result

SSL

O/JDBC REST

SSL SASL

SSL SSL SSL

LDAP

2.Knox Authenticates user/pass

Ranger Sync users/groups from LDAP

5. Ranger AuthZ

Apache Knox

Apache Knox


Kerberos

Page 8


-- Kerberos

20

Kerberos ID LDAP Active Directory ID

Kerberos Hadoop HortonworksOwen OMalley2010


Apache Ranger

Page 10


2014 Hadoop

Kerberos Apache Knox

Hive: ATZ-NG HDFS: ACLs Sentry SQL

Apache Falcon OS

2014 Hadoop


2014 5 Hortonworks XA Secure

+

+

+

+

+

2013 XA Secure Hadoop

Hortonworks XA Secure SQL Hdoop


Kerberos Apache Knox

HDFSHiveHBase

Apache Falcon OS

HDP HDP Hadoop

XA Secure HDP Hadoop HDFSHiveHBase

2014 Apache Software Foundation (ASF) Apache Ranger


Apache Ranger

Hadoop


HDFS Hive HBase Storm Knox REST API

HDFS, Hive, HBase, Storm, Knox

IP /


(HDFS)


(Hive)

17


18


Ranger

HDFS

Ranger Administration Portal

HBase

Hive Server2

Ranger Policy Server

Ranger Audit Server

Ranger Plugin

Had

oop

Com

pone

nts

Ent

erpr

ise

Use

rs

Ranger Plugin

Ranger Plugin

Legacy Tools & Data

Governance

Integration API

Oracle DB

HDFS

Knox

Kafka Ranger Plugin* Storm

Ranger Plugin

Ranger Plugin

MySQL

HDP 2.2 Additions Planned for 2015

Spark Ranger Plugin*

Solr Ranger Plugin*


HDP 2.2Ranger

Storm Knox

HDP Windows Hive Auth APIgrant/revoke HBase grand/revoke

REST APIs HDFS Oracle DB Ambari 2.0 Ambari


KnoxREST API Hadoop

Page 21


Data Lake

Hadoop API Hadoop REST API

REST APIs AuthN, AuthZ, SSL, SSO

REST APIs IdM , SSO, Oauth*, SAML*


Hadoop

Hadoop Cluster

Application Tier App A App N App B App C

Data Ingest

Data Share

Admin Operator Power User

Bastion Node

SSH

RPC Call

Falcon Oozie Scoop Flume

Data Operator

Business User

Hadoop System Admin

J/ODBC REST/HTTP

RPC


Load Balancer

KnoxHadoop API

Hadoop Cluster

Application Tier App A App N App B App C

Data Ingest

Data Share

Admin Operator Power User

Bastion Node

SSH

RPC Call

Falcon Oozie Scoop Flume Data

Operator

Business User

Hadoop System Admin

J/ODBC

REST/HTTP Knox


Knox

Kerberos API SSL

REST API SSH edge node

LDAP Active Directory SSO Apache Shiro

non-SSL SSL WebApp


Hadoop REST API with Knox

Service Direct URL Knox URL WebHDFS http://namenode-host:50070/webhdfs

https://knox-host:8443/webhdfs

WebHCat http://webhcat-host:50111/templeton

https://knox-host:8443/templeton

Oozie http://ooziehost:11000/oozie

https://knox-host:8443/oozie

HBase http://hbasehost:60080

https://knox-host:8443/hbase

Hive http://hivehost:10001/cliservice https://knox-host:8443/hive YARN http://yarn-host:yarn-port/ws https://knox-host:8443/resourcemanager

Masters could be on many

different hosts

One hosts, one port

Consistent paths

SSL config at one host


Hadoop REST API :

Pag 27

REST Client

Enterprise Identity Provider LDAP/AD

Knox Gateway

GW GW

Firewall

Firewall

DMZ

LB

Edge Node/Hadoop

CLIs RPC

HTTP

HTTP HTTP

LDAP

Hadoop Cluster 1 Masters

Slaves

RM

NN

Web HCat Oozie

DN NM

HS2

Hadoop Cluster 2 Masters

Slaves

RM

NN

Web HCat Oozie

DN NM

HS2

HBase

HBase


HDP 2.2Knox

Ambari

HDFS HAKnox YARN REST API Hadoop (WebHDFS, HBase, Hive & Oozie)SSL

Ranger for Knox Ranger Knox REST API


Page 29


HDP Hadoop3

? ?

, HDFS TDE, , OS , LUKS HDP, AES 256 for SSL & DTP

HDFS TDE, ,


:

Hadoop

HDFS TDE , GA 12015 Key Management Server TDE REST API

: Voltage, Protegrity, DataGuise LUKS

Volume Level Encryption (Open Source - LUKS, DMCrypt, Bit-Locker (Windows))

OS File Level Encryption (Open Source - eCryptfs)

Hadoop Level Encryption (HDFS TDE*, Hive CLE**, HBase** )

Custom Encryption Code

Partner (Voltage, Protegrity, Dataguise)

* - HDFS TDE ** - Future


RPC

Java SASL

DTP DN (3DES or RC4)

HTTP HTTP SSL

During Shuffle HTTPSMapperReducer

client

NameNode

DataNodes

Mapper Reducer

RPC request to R/W file

Block Data Transfer (DTP)

shuffle (HTTPS)


HDP

Page 33


HDFS

End to End

HiveServer 2 A B C

KDC

Use Hive ST, submit query

Hive gets Namenode (NN) service ticket

6.Hive creates MR/ Tez using NN ST

Ranger

3.Knox gets service ticket (ST) for Hive

4.Knox calls as proxy user

1.Original request w/user id/password

Client gets query result

SSL

O/JDBC REST

SSL SASL

SSL SSL SSL

LDAP

2.Knox Authenticates user/pass

Ranger Sync users/groups from LDAP

5. Ranger AuthZ

Apache Knox

Apache Knox


HDP 2.2Hadoop

Apache HadoopKerberos

Apache Knox Gateway HTTP/REST API

KnoxSSO

SSL & DTPAES

HDFS TDE ()

(Voltage, Protegrity, Data Guise)

HDP 2.2

HDFS, Hive, HBase, Knox, Storm


Thank You! Yifeng Jiang Solutions Engineer


Resources

Page 37


Security Page


Hortonworks Security Investment Plans

Comprehensive Security for Enterprise Hadoop

At Hortonworks.com/labs/security

Goals:

Investment themes

Central Administration Provide one location for administering security policies and audit reporting for entire platform

Comprehensive Security Meet all security requirements across Authentication, Authorization, Audit & Data Protection for all HDP components

Consistent Integration Integrate with other security & identity management systems, for compliance with IT policies

Ranger Phase Centralized Security Admin for HDFS, Hive & HBase

Centralized Audit Repor>ng Delegated Policy Administra>on

Previous Phases Kerberos Authen>ca>on HDFS, Hive & Hbase authoriza>on Wire Encryp>on for data in mo>on Knox for perimeter security Basic Audit in HDFS & MR SQL Style Hive Authoriza>on ACLs for HDFS

Delivered

Future Phases Encryp>on in HDFS, Hive & HBase Centralized security administra>on of en>re Hadoop plaJorm

Centralized audi>ng of en>re plaJorm Expand Authen>ca>on & SSO integra>on choices Tag based global policies (e.g. Policy for PII)

Delivered Ranger

HDP Security Overview

Technology

Transcript of HDP Security Overview