1. A TCloud Trender

2. WITHOUT KERBEROS Authorization Ensuring the user can only do things that they are allowed to do Yes: Owner/Group Permission Authentication Ensuring the user is who they claim to be NO 3. WITH KERBEROS 4. WITH KERBEROS 5. KERBEROS CONFIG BECOMING EASY Cloudera Cloudera Manager HDP Ambari Security Wizard --- ambari-1.2.5 6. HADOOP GATEWAYS client Firewall Gateway Hadoop Cluster 7. HADOOP GATEWAY - NOW Webhdfs Rest: curl "http://GATEWAYHOST/webhdfs/v1/PATH?[user.name=USER&]op= Hadoop: hadoop fs -fs webhdfs://GATEWAYHOST:14000 -cat FILe_PATH Oozie REST API , supports direct submission of MapReduce, Pig, and Hive jobs Steps Use webhdfs to upload your files and jars create an oozie workflow Hbase Hbase Stargate Rest Gateway Hbase Thrift server 8. HADOOP GATEWAY - FUTURE Apache Knox Gateway Provides a single point of authentication and access for Apache Hadoop services in a cluster 9. HADOOP GATEWAY - FUTURE Apache Knox Gateway Integrate with the existing frameworks for Active Directory /LDAP Shell and Rest Interface support Currently working on kerberized cluster support 10. HADOOP DATA ENCRYPTION Disk Encryption Partition Encryption dm-crypt File System Encryption Folder Encryption encryptfs Hadoop Encryption Framework Just encrypt what it should be 11. HADOOP ENCRYPTION FRAMEWORK - API Local File HDFS encrypt/ decrypt HDFS File Encrypt/decry pt 12. HADOOP ENCRYPTION FRAMEWORK - MR File Map File Reduce HDFS HDFS File Encryption/Decryption All the Path(Stages) 13. JIRAS hadoop-9331: Hadoop crypto codec framework and crypto codec implementations hadoop-9332: Crypto codec implementations for AES hadoop-9333: Hadoop crypto codec framework based on compression codec mapreduce-5025: Key Distribution and Management for supporting crypto codec in Map Reduce hbase-7544: Transparent table/CF encryption 14. Brief Two Crypto Typical Case in Hadoop Crypto API Case: Using AES Key (Store in KeyStore) to Encrypt/Decrypt Data MR CryptoContext Case: Encrypt the MR output Tool Distcrypto Hbase Encryption Other Related JIRAs and Security Key Store(Manager) TODOs 15. KEY STORE TOOL - KEYTOOL A key and certificate management utility. Create & Store an AES key keytool -keystore /tmp/hbase.jks -storetype jceks -storepass 123456 -genseckey - keyalg AES -keysize 256 -alias hbase Create & Store an RSA Private Key keytool -genkey -keyalg RSA -keysize 2048 -storetype jceks -storepass 123456 - keystore privateKeyStore.jks -alias testPrivate Export Certificate from KeyStore to a cert file keytool -export -keystore privateKeyStore.jks -storetype jceks -storepass 123456 - alias testPrivate -file publicKey.crt Import a cert file to a KeyStore keytool -import -trustcacerts -file publicKey.crt -storetype jceks -storepass 123456 - alias testPublic -keystore publicKeyStore.jks 16. CRYPTO API CASE: USING AES KEY (STORE IN KEYSTORE) TO ENCRYPT/DECRYPT DATA 17. CRYPTO API CASE: USING AES KEY (STORE IN KEYSTORE) TO ENCRYPT/DECRYPT DATA Use Crypto API to retrieve AES secret key from a key store file and use the key to encrypt/decrypt data KeyProvider CryptoContext CryptoCodec Sample Code 18. KeyProvider - KeyStoreKeyProvider To retrieve secret key from specified Key Store File Parameters keyStoreUrl & password keyStoreType: JCE, JCEKS keyPasswordFile & sharePassword Initial: keyProvider.init(String parameters) String parameters = file:///etc/keystore/mapred.jks&keyStoreType=JCEKS &password=123456 String parameters = KeyStoreKeyProvider.getKeyStoreParameterString(fileUrl, StoreType, StorePassword, null, true); Get: keyProvider.getKeys(String []) 19. CryptoContext To store key related info Key Attributes Raw Key Data Key Type: SYMMETRIC_KEY, PUBLIC_KEY, PRIVATE_KEY, CERTIFICATE Cryptographic Algorithm: e.g AES Cryptographic Length 20. CryptoCodec A wrap, contain CryptoContext and provide Crypto IO Stream Major member CryptoContext Crypto IO Stream Method createOutputStream() createInputStream() 21. SAMPLE CODE --- FILE ENCRYPTION 22. SAMPLE CODE --- FILE ENCRYPTION - conti 23. MR CRYPTOCONTEXT CASE: ENCRYPT THE MR OUTPUT 24. MR CRYPTOCONTEXT CASE: ENCRYPT THE MR OUTPUT Using provided CryptoContext to encrypt the Map Reduce output KeyProviderConfig CryptoContextProvider Sample Code 25. KeyProviderConfig Members keyProvider keyProviderParameters Methods getKeyProvider() getKeyProviderParameters() 26. CryptoContextProvider Provide several static helper methods to update Crypto related Job Configurations. For example, store the following Parameters and Secrets to the Job Credential in the secret key list mapred.[[[STAGE]]].crypto.context.provider.parameters mapred.[[[STAGE]]].crypto.context.secrets [[[STAGE]]]: input, output, map.output AbstractCryptoContextProvider FileMatchCryptoContextProvider KeyProviderCryptoContextProvider Credentials credentials = jobConf.getCredentials(); credentials.addSecretKey(new Text("mapred.map.output.crypto.context.provider.parameters"), parameters); credentials.addSecretKey(new Text("mapred.map.output.crypto.context.secrets"), secrets); 27. AbstractCryptoContextProvider Methods setInputCryptoContextProvider (jobConf, cryptoContextProviderClass, secrets, parameters) setMapOutputCryptoContextProvider (jobConf, cryptoContextProviderClass, secrets, parameters) setOutputCryptoContextProvider (jobConf, cryptoContextProviderClass, secrets, parameters) 28. FileMatchCryptoContextProvider Provides the ability to select the appropriate CryptoContext according to the file path FileMatches fileMatches = new FileMatches(KeyContext.derive("12345678")); fileMatches.addMatch("^.*/input1.intel_aes$", KeyContext.derive("1234")); fileMatches.addMatch("^.*/input2.intel_aes$", KeyContext.derive("5678")); FileMatchCryptoContextProvider.setInputCryptoContextProvider(jobConf, fileMatches, null); 29. KeyProviderCryptoContextProvider Not only include the capability of FileMatchCryptoContextProvider also provide the ability to retrieve the Key from Key Store FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00", Key.KeyType.SYMMETRIC_KEY, "AES", 128)); String keyStoreFile = "file:///" + KEYSTORE_HOME + "/mr.jks"; String keyStorePassword = "12345678"; KeyProviderConfig keyProviderConfig = KeyProviderCryptoContextProvider.getKeyStoreKeyProviderConfig( keyStoreFile, "JCEKS", keyStorePassword, null, true); KeyProviderCryptoContextProvider.setInputCryptoContextProvider(jobConf, fileMatches, true, keyProviderConfig); 30. SAMPLE CODE - ENCRYPT THE MR OUTPUT Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); JobConf jobConf = (JobConf)job.getConfiguration(); 31. SAMPLE CODE - ENCRYPT THE MR OUTPUT Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); JobConf jobConf = (JobConf)job.getConfiguration(); FileOutputFormat.setOutputCompressorClass(job, AESCodec.class); jobConf.set(AESCodec.CRYPTO_COMPRESSOR, org.apache.hadoop.io.compress.SnappyCodec); 32. SAMPLE CODE - ENCRYPT THE MR OUTPUT - Conti FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00", Key.KeyType.SYMMETRIC_KEY, "AES", 256)); 33. SAMPLE CODE - ENCRYPT THE MR OUTPUT - Conti FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00", Key.KeyType.SYMMETRIC_KEY, "AES", 256)); String keyStoreFile = "file:///" + KEYSTORE_HOME + "/mr.jks"; String keyStorePassword = "12345678"; KeyProviderConfig keyProviderConfig = KeyProviderCryptoContextProvider.getKeyStoreKeyProviderConfig( keyStoreFile, "JCEKS", keyStorePassword, null, true); 34. SAMPLE CODE - ENCRYPT THE MR OUTPUT - Conti FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00", Key.KeyType.SYMMETRIC_KEY, "AES", 256)); String keyStoreFile = "file:///" + KEYSTORE_HOME + "/mr.jks"; String keyStorePassword = "12345678"; KeyProviderConfig keyProviderConfig = KeyProviderCryptoContextProvider.getKeyStoreKeyProviderConfig( keyStoreFile, "JCEKS", keyStorePassword, null, true); KeyProviderCryptoContextProvider.setOutputCryptoContextProvider(jobConf, fileMatches, false, keyProviderConfig); . job.waitForCompletion(true); 35. MORE IN KeyProviderCryptoContextProvider Using asymmetric key (RSA) to protect Parameters & Secrets 36. MORE IN KeyProviderCryptoContextProvider Using asymmetric key (RSA) to protect Parameters & Secrets CredentialProtection credentialProtection = new CredentialProtection(jobConf, RSACredentialProtectionCodec.class, encryptionKeyProviderConfig, encryptionKeyName, decryptionKeyProviderConfig, decryptionKeyName); KeyProviderCryptoContextProvider.setInputCryptoContextProvider( jobConf, fileMatches, false, keyProviderConfig, credentialProtection); 37. MORE IN KeyProviderCryptoContextProvider - Conti How to use Customized KeyProvider in KeyProviderCryptoContextProvider String keyProviderParameters = KeyStoreKeyProvider.getKeyStoreParameterString( keyStoreFile, keyStoreType, keyStorePassword, keyStorePasswordFile, sharedPassword); KeyProviderConfig keyProviderConfig = new KeyProviderConfig( CustomizeKeyStoreKeyProvider.class.getName(), keyProviderParameters); 38. TOOL DISTCRYPTO Use MapReduce Job to encrypt, decrypt or key rotate multiple files Usage -op : "encrypt, "decrypt" and "rotate --ek -dk -src -dst -log 39. TOOL DISTCRYPTO - conti Source Definition File (XML format) src path format: raw Sequence the full class name of a class which implement CryptoHandler for customized format. includeFilter & excludeFilter stripSuffix & appendSuffix keyClassName & valueClassName. 40. TOOL DISTCRYPTO - conti Encryption Sample command hadoop distcrypto -op encrypt -ek 21EF7D7487F69A19E552C1274A9FCAC721EF7D7487F69A19E552C1274A9F CAC7 -log /tmp/log.distcrypto.encrypt -src file:///working/crypto_encrypt.xml Source Definition File (crypto_encrypt.xml) TODO: Not support retrieve keys from key store --- Not Good /tmp/install.lograw.encrypted 41. HBASE ENCRYPTION HBASE-7544 42. HBASE ENCRYPTION HBASE-7544 Introduce transparent encryption of HBase on disk data. Transparent encryption at the CF level Two-tier key architecture for consistency with best practices for this feature in the RDBMS world Flexible and non-intrusive key rotation 43. HBASE ENCRYPTION HBASE-7544 44. HBASE ENCRYPTION HBASE-7544 HFile Block0 Block N Meta Block0 Meta Block N File Info Data Block Index Mwta Block Index Fixed File Trailer Key block data format 1 byte ordinal 4 bytes key data length encrypted key data Encryption KeyBlock Offset 45. HBASE-7544 SETTINGS 1. Set up the keystore with a secret key Create a secret key of appropriate length for AES. $ keytool -keystore /path/to/hbase/conf/hbase.jks-storetype jceks -storepass password-genseckey -keyalg AES -keysize 256-alias ${USER} Press RETURN to store the key with the same password as the store 46. HBASE-7544 SETTINGS 2. Configure HBase to use the keystore Add this to the hbase-site.xml file: hbase.crypto.keyproviderorg.apache.hadoop.io.crypto.KeyStoreKeyProviderhbase.crypto.keyprovider.parameterskeyStoreUrl=file:///path/to/hbase/conf/ hbase.jks&keyStoreType=JCEKS&password=password 47. HBASE-7544 SETTINGS 3. Create the table $ ./bin/hbase shell hbase(main):001:0> create 'test', {NAME=>'t', CRYPTO=>'AES', CRYPTO_KEY=>'123456'} 48. HBASE-7544 CF key rotation CF key is changed by modifying the column descriptor via HBaseAdmin. Then, major compaction is triggered either on the table at once or region by region. Performance Using this AES-NI codec, HFile read and write code paths introduces an overhead roughly on par with GZIP compression for reads, and half that as for writes. 49. OTHER RELATED JIRAS MAPREDUCE-4491: Encryption and Key Protection 4550: Key Protection : Define Encryption and Key Protection interfaces and default implementation 4551: Key Protection : Add ability to read keys and protect keys in JobClient and TTS/NodeManagers 4552: Encryption: Add support for PGP Encryption 4553: Key Protection : Implement KeyProvider to read key from a WebService Based KeyStore 5025: Key Distribution and Management for supporting crypto codec in Map Reduce 50. SECURITY WEB KEYSTORE SERVER safe (http://benoyantony.github.com/safe/) Web service based keystore Support ACL Per Key Authenticates the user using SPNego Base on Cloudera Alfredo, a Java library consisting of a client and a server components to enable Kerberos SPNEGO authentication for HTTP. WEB Server (safe(alfredo)) KDC user authorization authentication MR/Hbase + WebStoreKeyProvider 51. OTHER TODOs Hive support https://issues.apache.org/jira/browse/HIVE-5207 Support data encryption for Hive tables https://issues.apache.org/jira/browse/HIVE-4227 Add column level encryption to ORC files Created: 25/Mar/13 17:14 Pig support https://issues.apache.org/jira/browse/PIG-3289 Encryption aware load and store functions 52. Q & A