Farmacologiadelaparatodigestivodiaspositivas3 150201000849 Conversion Gate01
Teradata10162014stratapresentation 141028143551 Conversion Gate01
description
Transcript of Teradata10162014stratapresentation 141028143551 Conversion Gate01
Teradata and Hortonworks The Unified Data Architecture (UDA)
16th October, 2014
2
Shift from a Single Platform to an Ecosystem
“The hype around replacing the data warehouse gives way to the more sensible strategy of augmenting it … The influence of the logical data warehouse has created a situation in which multiple repository strategies are now expected.”
"Logical" Data Warehouse
“Big Data requirements are solved by a range of platforms including analytical databases, discovery platforms, and NoSQL solutions beyond Hadoop.”
Source: “Big Data Comes of Age”. EMA and 9sight Consulting. Nov 2012.
Math and Stats
Data Mining
Business Intelligence
Applications
Languages
Marketing
ANALYTIC TOOLS & APPS
USERS
INTEGRATED DISCOVERY PLATFORM
INTEGRATED DATA WAREHOUSE
ERP
SCM
CRM
Images
Audio and Video
Machine Logs
Text
Web and Social
SOURCES
DATA PLATFORM
ACCESS MANAGE MOVE
UNIFIED DATA ARCHITECTURE System Conceptual View
Marketing Executives
Operational Systems
Frontline Workers
Customers Partners
Engineers
Data Scientists
Business Analysts
Math and Stats
Data Mining
Business Intelligence
Applications
Languages
Marketing
ANALYTIC TOOLS & APPS
USERS
INTEGRATED DISCOVERY PLATFORM
INTEGRATED DATA WAREHOUSE
ERP
SCM
CRM
Images
Audio and Video
Machine Logs
Text
Web and Social
SOURCES
DATA PLATFORM
Business Intelligence
Predictive Analytics
Operational Intelligence
Data Discovery
Path, graph, time-series analysis
Pattern Detection
Fast Data Loading & Availability
Filtering & Processing
Deep History: Online Archival
UNIFIED DATA ARCHITECTURE Business Conceptual View
Fast-Fail Hypothesis Testing
Marketing Executives
Operational Systems
Frontline Workers
Customers Partners
Engineers
Data Scientists
Business Analysts
ACCESS MANAGE MOVE
Data Mgmt. (data lake)
5
Discovering Deep Retail Insights with UDA Transforming Web Walks into DNA Sequences
Impact
• Leverage Aster platform to generate rapid path insights • Drives 15% increase in market baskets through personalization • Drives 10-20% increase in conversions by shortening paths • Can now see what does and doesn’t lead to sales • Widening use across all the Corporate Group websites
Situation
Largest German online retailer, conglomerate with numerous brands and 50 websites. 1 Millions visitors, viewing 2M products.
Problem
Needed a better way of analyzing consumer behavior on the websites, communicating with category managers
Solution
Treat each web visit sequence like DNA sequence. Built a fast query tools so analysts can express queries easily for their categories, get deeper insights
KNOX
AMBARI
SOURCE DATA
Sensor Log Data
Customer/Inventory
Data
Clickstream Data
Flat Files
Sentiment Analysis
Data
DB
File
JMS
REST
HTTP
Streaming
Analytical Platforms
Teradata IDW
Aster Discovery Platform
Query/Visualization/ Reporting/Analytical
Tools and Apps
JDBC/ODBC Compliant Tool
MAPREDUCE YARN
Viewpoint Alerts Services System
Health Node
Health Space Usage
Capacity Heatmap
Metrics Analysis
TVI – Proactive system monitoring tied to Teradata customer support
HDFS
REFINE HIVE
PIG
CUSTOM
ETL
LOAD SQOOP
FLUME
Web HDFS
NFS EXTRACT
STRUCTURING
HCATALOG
INTERACTIVE
QueryGrid
EXPORT SQOOP / HIVE
LOAD TDCH
BULK COPY
DISTCP AFS
EXTRACT
Modern Data Architecture: Teradata
Bidirectional
7
• Most Trusted and Flexible Hadoop Platforms for Your Next-Generation Unified Data Architecture™
1. Teradata Aster Big Analytics Appliance
2. Teradata Appliance for Hadoop
3. Teradata Commodity Offering with Dell
4. Hortonworks Data Platform software-only support resell
• Complete consulting and training capability
> Big Analytics Services—across the UDA
> Data Integration Optimization—ETL, ELT across the UDA
> Hadoop deployment and mentoring
> Teradata delivering Hortonworks training
> Hadoop Managed Services—operations and administration
• Customer Support for Hadoop > World-class Teradata customer support, backed by Hortonworks
Teradata Portfolio for Hadoop ” Bringing Hadoop to the Enterprise”
8
Loom is a platform for profiling, preparing and tracking data lineage for data in Hadoop
• Hadoop Data Governance and Metadata Management – Rich information model for capturing and managing the relationships – Data dictionary for the big data landscape – Support for non-Hadoop sources
• Automation (Activescan) – Discovering and introspecting new data in the cluster – Triggering external processing (e.g. Oozie script for ETL) – Automatically collecting metadata about the job - lineage, statistics – Polling YARN job history for lineage
• User Interactivity (Workbench) – Advanced user interfaces for data exploration, profiling and preparation – Data wrangling for interactively cleaning/reshaping raw data into useable data
Teradata Loom® 2.3 “Integrated metadata management, data lineage
and data wrangling for Enterprise Hadoop”
Free version of Loom pre-installed with Hortonworks Sandbox
9
Teradata Appliance for Hadoop
Optimized hardware for Hadoop
BYNET™ V5 40GB/s InfiniBand interconnect
Tera
da
ta V
ital I
nfr
ast
ruc
ture
Teradata Distribution for Hadoop (Based on Hortonworks HDP)
NameNode Failover
Intelligent Start and Stop
Teradata Connector for Hadoop (TDCH)
Teradata QueryGrid ® Teradata Studio with
Smart Loader
Teradata Viewpoint
Value Added Software from Partners
HCatalog
Kerberos
Teradata Loom® ( for data management )
10
Teradata QueryGrid™ Vision
TERADATA ASTER
DATABASE
SQL, SQL-MR, SQL-GR
Multiple Teradata Systems
TERADATA DATABASE
HADOOP
Push-down to Hadoop
System
IDW
TERADATA DATABASE
Discovery
TERADATA ASTER
DATABASE
Business users Data Scientists
COMPUTE CLUSTER
Run SAS, Perl, Ruby, Python, R
RDBMS DATABASES
Push-down to Other
Database
MONGODB DATABASE
Push-down to NoSQL
Databases
11
• Trusted: Use existing tools/skills and enable self-service BI with granular security
• Standard: 100% ANSI SQL access to Hadoop data
• Fast: Queries run on Teradata or Aster, data accessed from Hadoop
• Efficient: Intelligent data access leveraging the Hadoop HCatalog
Hadoop Layer: HDFS
Pig
Hive
Hadoop MR
QueryGrid: Teradata-Hadoop QueryGrid: Aster-Hadoop
HCatalog
Da
ta
Da
ta F
ilte
ring
Give business users on-the-fly access to data in Hadoop
Teradata QueryGrid™: Teradata - Hadoop
12
Teradata Viewpoint
• Hadoop Portlets: – Node Monitor (Aster & Hadoop)
– Hadoop Services
• Integration into existing: – Monitoring: System Health, Metrics
Analysis, Metrics Graph, Capacity Heatmap, Space Usage.
– Admin: Alert Viewer, Alert Setup, Teradata Systems, Role Manager
Single Operational View (SOV) for Teradata, Aster, & Hadoop
13
• Key Features – High-speed connector between Teradata and
Hadoop based on Apache Sqoop framework
– Both import and export data between Teradata and Hadoop
– Leverages the JDBC-FastLoad/FastExport mechanism from Teradata
– Import/export Hive rcfile/sequencefile/textfile format and Hive partitioned files
Teradata Connector for Hadoop (TDCH)
INTEGRATED DATA WAREHOUSE
CAPTURE | STORE | REFINE
• Available through Hortonworks > Hortonworks
• Teradata Connector for Apache Hadoop (Release v1.2.0) • Download link: http://hortonworks.com/download/
14
• Hadoop View – Browse through tables
within the Hadoop cluster - Views table properties
– Bi-directional table copies - Drag and drop interface
- Maps data types between Hadoop and Teradata tables
– Transfer Status and History - Track load status
• Benefits – Simplifies Hadoop browsing
– Ad hoc data movement between Teradata and Hadoop
– No scripting required
– Point and click
Teradata Studio: Smart Loader for Hadoop Self-Service Load
15
Questions and Next Steps
More about Teradata & Hortonworks http://www.hortonworks.com/partner/teradata/
Teradata Loom for HDP http://www.teradata.com/tryloom
Find Us @Strata
Booth # 324 Teradata Hadoop Station