StreamCentral for the IT Professional
-
Upload
raheel-retiwalla -
Category
Technology
-
view
412 -
download
2
description
Transcript of StreamCentral for the IT Professional
http://www.virtus-it.comA trusted partner
Software to model & build Business Intelligence & Big Data Solutions
http://www.virtus-it.com 2
Table of Contents
• Big Data Overview………..………………………………………………………… 3 - 6• Step by Step approach to building Big Data Solutions……………….. 7 - 10• StreamCentral Introduction……………………………………..………….... 11 - 21• Keeping it structured? –Extending current DW/BI investments………. 22 - 25• An approach to building Operational Intelligence solutions………….... 26 - 28• StreamCentral additional details……………………………………..………….... 29 - 37• StreamCentral physical architecture……………………………..………….... 38 - 40• How StreamCentral fits in an enterprise technology architecture…… 41 - 44
http://www.virtus-it.com
10101010101010101010101010101010101010101010101ABC0101010101010101010101010100101010101010101ABCABCABCABC
101010
101010
3
BIGDataToday With Big Data
Custom Application
ERP
• Analysis of structured data from internal applications
• Data sets updated using batch processes
• Traditional BI & Data Warehousing
• Traditional BI and data warehousing extended to include structured and unstructured data from internal and external applications processed in real-time or in batch.
• Ability to predict events as well as analyze historical associations in wide sets of data for patterns and trends.
Data Management Discovery & Analysis Event DetectionBig Data Solution
Modeling & Model Deployment
Stream Processing &Batch Data Acquisition
Blocks for building Big Data Solutions
ERP
Internal & External Applications Data Stores
101010
Real-Time + Batch Big Data Processing Layer
Real-time event data in operational applications
Pattern, trend and association analysis
Understand the connections in a wide variety of data that impact business performance and use that knowledge to deliver exceptional business results
http://www.virtus-it.com 4
Understanding business performance with Big Data includes two distinct capabilities:
Managing performance by analyzing internal and external, structured and unstructured data for patterns and associations collected over time
• Customer segmentation based on buying history patterns and finding associations with population, census and twitter data to develop marketing strategy
• Web analytics to improve marketing campaigns and relevant content
• Sales pipeline analysis compared to industry data to understand the right goals to set
• Cash flow analysis to make capital investment decisions considering external variables
Managing performance by analyzing real-time data for day to day events – Operational Intelligence
• What is the current workload?• Is staff available and working on
high priority work?• What factors are impacting
customer experience right now?• What processes are taking longer
than expected?
A few example scenarios: A few example scenarios:
http://www.virtus-it.com 5
Telco’s Core IMS Network Data Data, Voice & Video
Performance DataData, Voice & Video Performance Data
Data from Telco Towers
Weather DataTraffic
IncidentsPopulation Data
Data Stream
weatherunderground
MapquestUSA Today Census data
Sources of real time streaming data from networks, devices, services and other internal applications
External sources of data that add understanding of what’s happening when events are detected
Example Big Data Solutions: Telco
Network Test
New Service –
Investment Planning
Adaptive Bit Rate –
Video Streaming
QoE
360o Customer QoE for 1st Level
customer service
Video QoE for IPTV
Business Solutions
http://www.virtus-it.com 6
Telco’s Core IMS Network Data
Data, Voice & Video Performance Data
Data, Voice & Video Performance Data
Data from Telco Towers
Weather Data Traffic IncidentsPopulation Data
Data Stream
weatherunderground INRIXUSA Today Census data
Example Big Data Scenario : Utilities - Water
Sources of real time data relating to your business
Sources of BIG DATA relevant to your business
VIBRATION SENSOR
ENERGY HARVESTING
WATER MAIN PRESSURE
SCADA NETWORK
http://www.virtus-it.com 7
Steps it takes to build powerful Big Data solutions!
Solution Modeling
Model Deployment
Data Acquisition
(streaming or batch, internal or external, structured or unstructured)
Data Management
Event Detection
Discovery & Analysis
Big Data Solution Lifecycle
Start here
http://www.virtus-it.com 8
Solution Modeling
• Logical Data Model design
• Data standardization & transformation modeling
• Key Performance Indicator modeling via business rules
• Dimensional modeling
• Historical Data Mart Modeling
• Event detection modeling via business rules
• Real-time analytics data mart modeling
Model Deployment
• Physical Design Implementation
• Physical deployment of dimensional model
• Database deployment
• Physical deployment of data marts
• Rules deployment
Data Acquisition
• Data from internal data sources
• Data from external sources
• Streaming data• Batch data• Structured Data• Unstructured Data• Data transformation• Data standardization
Data Management
• Structured Data Storage
• Unstructured Data Storage
• Scalability• Performance
Event Detection
• Detecting events on streaming data
• Alerting• Integration with
operational applications
Discovery & Analysis
• Information Discovery
• Data Classification• Analytics• Querying• Visualization
Solution Modeling
Model Deployment
Data Acquisition
Data Management
Event Detection
Discovery & Analysis
Big Data Solution Lifecycle – Tasks Detailed
http://www.virtus-it.com
9
Solution Modeling
Model Deployment
Data Acquisition
Data Management
Event Detection
Discovery & Analysis
1. Hadoop - MapReduce2. MPP Columnar Databases like Neteeza, Vertica, ParStream3. NoSQL – MongoDB, Cassandra4. Evolution of traditional RDBMS to support column indexes –
SQL Server
Big Data Innovations in Data Management
Big Data Innovations in Discovery & Analysis
Where has the innovation been in Big Data?The last few years have seen lots of innovation in Data Management as well as Discovery and Analysis
http://www.virtus-it.com 10
Solution Modeling
Model Deployment
Data Acquisition
Data Management
Event Detection
Discovery & Analysis
Big Data Lifecycle
But, where is the innovation in these areas?
• Fragmented, point use or lack of industry strength technology to aid in Design, Model Deployment, Data Acquisition and Event Detection makes it difficult, time consuming and specialist resource intensive to build Big Data Solutions
• What is the use of having scalable platforms that can store and manage this data and tools that can deliver incredible visualizations when the effort to get the data right is still a problem as it has always been?
http://www.virtus-it.com 11
Introducing StreamCentral
http://www.virtus-it.com 12
Solution Modeling
Model Deployment
Data Acquisition
Data Management
Event Detection
Discovery & Analysis
Big Data Solution Life cycle
1. StreamCentral Solutions Designer makes it easy to model traditional BI/DW and Big Data solutions
2. Builds and deploys model on HP Vertica or Microsoft SQL Server
3. Adds context by connecting all streaming and static data to time, location and entities
4. StreamCentral Big Data Server, horizontally scalable, executes the model definition in real-time
5. StreamCentral drastically reduces time to market, risk and cost in building Big Data solutions!
Software to design & build BI & Big Data SolutionsStreamCentral enables you to quickly move from a blank
sheet of paper to a production system, comprehensive and powerful that can be delivered without a large investment in specialist skills.
http://www.virtus-it.com 13
1010101010101010ABCABCABCABCABC
StreamCentral Workbench: Solution Designer
StreamCentral Workbench:
Model DeploymentData Collection
Data Processing Correlation Data
PublishingData
Security
StreamCentral Big Data Server
StreamCentral has three main components:1. Use the Workbench Designer to define source data, entities,
rules for monitoring conditions, events and data correlation, analytical models and knowledgebase
2. Workbench Model Deployment configures, builds and deploys the model on top of HP Vertica or Microsoft SQL Server
3. Big Data Server executes the defined model in real-time
1
2
3
http://www.virtus-it.com
DatabaseREST/SOAP
API LDAP
PUSH
API
Data Processing Engine
Vertica SQL Server
Correlation EngineCollector Data Publishing, Access and Security
• Capture data• Validate data• Prepare data
• Apply transformations• Perform calculations• Determine conditions & KPIs• Identify & build dimensions• Identify alerts
• Correlate incoming data based on defined rules
• Detect events based on correlated data
• Update fact data• Update entity & dimension data• Update analysis collections• Update event collections• Manage data level security
Data Acquisition – Push / Pull data from
variety of sources
Design data transformations
Conditions & KPI modeler via rules
builder
Real-time data correlation
Event detection via rules builder
Real-time data mart designer
360o data mart designer
Define entities and Import Entity Data
Dimension modeler
Data Security designer
StreamCentral Big Data Server
StreamCentral Workbench: Big Data Solution Designer
Meta DataCreate Database
Structure Add Context
StreamCentral Workbench: Big Data Solution Deployment
http://www.virtus-it.com 15
Model Pull Data
Sources with strong REST, SOAP & DB
Support Push Data API
Data Transformat-
ion
Model Entities &
import static data
Dimension modeler
Time & Location
Standard-ization
Conditions & KPI
modeler
Correlation Modeler
Event Detection rules on real-time
data
Real-time & Historical analytics
Data mart modeler
• Software targeted to be used by IT and non IT people to design and build Big Data solutions
• Can work with batch data (as in traditional Business Intelligence) or real-time streams (as in Operational Intelligence)
• Workbench lets analysts model all necessary steps in building a Big Data Solution• Data Pull/Push• Model Transformations• Model Entities (like customers, patients, products),
import static entity data and define entity relationships to source data
• Shared dimensions across data• Condition modeler via business rules to monitor specific
sets of conditions in batch or streaming data• Evaluate different entities with different sets of conditions
as data flows in• Specify rules to model how to correlate data streams in
real-time• Event detection• Model data marts that aggregate the right data for
association and pattern analysis
StreamCentral Workbench : Software to design traditional BI/DW & Big Data Solutions
Workbench
http://www.virtus-it.com 16
Generating insights from data requires context to be added to the data. This context is a continuous thread that connects all types of data throughout the Big Data Solution lifecycle. Four typical examples of context..
Insight
Who (entities like customer,
patient)
When (time) Where (location)
What (streaming &
static data correlation)
• StreamCentral automatically builds and maintains time and location dimensions
• Entities can be created and defined in StreamCentral
• All data in StreamCentral is continuously and automatically connected to time, location and defined entities
• Resultant real-time events and analytical data marts automatically inherit this context without need for any programming or development work
• This increases the impact and value of collected data
Converting data to insights by continuously adding context
http://www.virtus-it.com 17
Auto build and deploy
DB structure based on
Workbench Model
Continuous Pull with
strong REST, SOAP & DB
Support
Push Data API for
streaming sources
Time & Location
Standard-ization
Monitor conditions
Event detection
Build data marts &
continuously update new
data
In-Memory Operations
Distributed Architecture
MPP Support
StreamCentral Big Data Server: Software that runs Big Data Solutions
• Extends your Business Intelligence strategy by easily incorporating external data sets
• Introduces integration of real-time data for event insight to your organization
• Auto-builds database schema (facts, dimensions, entities, flat tables and more)
• By default, standardizes all incoming data by connecting it to auto created time and location dimensions
• Builds event data marts and continuously loads data
• Builds real-time data marts to help in understanding associations in data Continuously loads these analysis data marts
• Deliver real-time event insights to new or existing operational applications
• Significantly reduces IT overhead in building Big Data solutions
Big Data Server
http://www.virtus-it.com
18
Solution Modeling
Model Deployment
Data Acquisition
Data Management
Event Detection
Discovery & Analysis
Bringing it together: Building Big Data Solutions with StreamCentral and partner solutions
1. MPP Columnar Databases : Vertica, ParStream
2. Microsoft SQL Server
StreamCentral Big Data Server
StreamCentral Workbench: Model Deployment
StreamCentral Workbench: Big Data Solutions Modeler
Tableau Software, Microsoft PowerView
StreamCentral Big Data Server
http://www.virtus-it.com
• Industrial strength, enterprise ready with web scale characteristics - handles extremely large amounts of data
• Uses in-memory processing for high speed• Next generation distributed architecture – allows you to
run on any number of commodity hardware • Built in redundancy at every layer for high availability• Easy to use tools to monitor and manage StreamCentral• Built on Microsoft technology that most enterprises
already have invested in• Runs on best of breed and latest database technology
from Microsoft SQL Server and HP Vertica
Choose database from:
19
http://www.virtus-it.com 20
Why StreamCentral?• Roadmap to Big Data: StreamCentral is the only solution that enables the evolution of current
practices in Business Intelligence and Data warehousing to now include external data, event monitoring and real-time insights
• No programming solution modeler: StreamCentral takes a solution approach – designing and modeling shifts to analysts versus everything being done by developers or programmed from scratch
• Continuously adds context to data: Any kind of data that is streamed to StreamCentral, pulled in near real-time or imported via batch is continuously and automatically connected to time, location and defined entities. This significantly reduces risk, time and cost associated with building BI/DW and Big Data solutions
• Reduced dependency on specialist skills: No in-depth knowledge needed on HP Vertica or SQL Server development as StreamCentral builds, deploys and maintains all internal structures in those environments automatically
• Plays well: Is standards based and agnostic to existing enterprise technologies• Adaptable: Everything created in StreamCentral can be modified. Makes it easy to adapt the Big
Data solution to changing needs of the business
http://www.virtus-it.com 21
Making a business case for leveraging Big Data just got a whole lot easier with StreamCentral
70%Time taken to build Big Data
solutions is drastically reduced by using StreamCentral
60%Cost of building Big Data
solutions is drastically reduced by using StreamCentral
In addition, StreamCentral reduces risk, data quality issues, specialist skillsets requirements and complexity in building traditional Business Intelligence/Data Warehousing or Big Data solutions
http://www.virtus-it.com 22
No immediate plans to go Big on Data? Planning to work primarily with structured data?
But would like to deliver additional insights by enhancing your existing investments in Business Intelligence and Data Warehousing?
http://www.virtus-it.com 23
Traditional Data WarehousingInterrogation of historical data for trend analysis. Business Intelligence applications deliver analytics or reports to management for performance analysis
On-Demand Business IntelligenceUpdate Data Warehouse continuously with real-time data. Provides the ability to analyze data updated in real-time
Operational Intelligence
Allows organizations to monitor fast moving data for key indicators and events and immediately act on these insights, through manual or automated actions
Reporting:-What did happen ?
Analysis:- Why did it happen ?
Happens on previously stored data (data at rest) Happens on real-time streaming data (data in-flight)
Solution value to businessLower Higher
Perc
eive
d Co
mpl
exity
Hig
her
Low
er
Event Monitoring:- What is happening ?
Predictive Analytics:-What will happen ?
Traditional Data Warehousing Solutions
On-Demand BI
OperationalIntelligence
Keeping it structured – A roadmap to extend current investments in BI/DW
http://www.virtus-it.com
Reporting:-What did happen ?
Analysis:- Why did it happen ?
Happens on previously stored data (data at rest)
Happens on real-time streaming data (data in-flight)
Solution value to businessLower Higher
Perc
eive
d Co
mpl
exity
Hig
her
Low
er
Event Monitoring:- What is happening ?
Predictive Analytics:-What will happen ?
Traditional Data Warehousing Solutions
On-Demand BI
Operational Intelligence
Most organizations have traditionally invested in this area
In most companies, the scope of understanding business performance is limited to historical analysis and rarely includes real-time understanding of key events that impact day to day operational processes
Keeping it structured – A roadmap to extend current investments in BI/DW
http://www.virtus-it.com 25
Reporting:-What did happen ?
Analysis:- Why did it happen ?
Happens on previously stored data (data at rest)
Happens on real-time streaming data (data in-flight)
Solution value to businessLower Higher
Perc
eive
d Co
mpl
exity
Hig
her
Low
er
Event Monitoring:- What is happening ?
Predictive Analytics:-What will happen ?
Traditional Data Warehousing Solutions
On-Demand BI
Operational Intelligence
Most organizations have traditionally invested in this area
StreamCentral’s area of focus
Keeping it structured – A roadmap to extend current investments in BI/DW
http://www.virtus-it.com 26
An approach to working with real-time data -
Operational Intelligence
http://www.virtus-it.com 27
Data Layer
Interfaces
Data Processing
Real-Time Insights
Business Solutions
Operational (User)
Internal Applications and Data Sets
External Data
Connections to existing architecture for tapping data & data streams
APIsDatabases
Enterprise Service Bus
Messages
Push Streaming Data |Pull Data |Format | Standardize | Transform | Measure | Correlate | Event Detection | Rules Engine | In-Memory Processing Real-Time Streaming Analytics
Real-Time Event NotificationHistorical data that supports pattern &trend analytics. New insights are added in real time
CustomerExperience
ContinuousImprovement
Day to day insights and actions delivered in multiple mediums to many users
KPIsComplaintsBrand –Protection
1
2
3
4
5
6
!
Access to right information at the right time along with knowledge base of actions to perform
Operational Intelligence practices are similar to traditional Data Warehousing practices
http://www.virtus-it.com 28
Data Layer
Interfaces
Data Processing
Real-Time Insights
Business Solutions
Operational (User)
Internal Data Sets External Data
Connections to existing architecture for tapping data & data streams
APIsDatabases
Enterprise Service Bus
Messages
Push Streaming Data |Pull Data |Format | Standardize | Transform | Measure | Correlate | Event Detection | Rules Engine | In-Memory Processing Real-Time Streaming Analytics
Real-Time Event NotificationHistorical data supporting pattern &trend analytics. New insights added in real time
CustomerExperience
ContinuousImprovement
Day to day insights and actions delivered in multiple mediums to many users
KPIsComplaintsBrand –Protection
1
2
3
4
5
6
!
Access to right information at the right time along with knowledge base of actions to perform
Focus of StreamCentral
http://www.virtus-it.com 29
More details on how StreamCentral works
http://www.virtus-it.com 30
StreamCentral Workbench Big Data Solutions Modeler - Inputs
• Data Sources• Push/Pull• Data transformations
• Define and import entity data• Modeling
• Rules for monitoring conditions in data• Correlation rules to identify related records across data sources in real-time• Rules for detecting events• Common dimension modeling• Data Mart modelers
• Support for Real-time• Correlation rules to identify related records across data sources in real-time• Rules for detecting events• Configure real-time data marts
• 360o data aggregation**• Define data relationships across data sources• Configure 360o data marts
• Data level security**** Coming Q3 2013
http://www.virtus-it.com 31
StreamCentral Big Data Server - Output• Database structure automatically created, updated and managed in Big Data databases like HP
Vertica or SQL Server by StreamCentral.
• The StreamCentral database automatically builds time and location dimensions, fact tables, other dimension tables, standardizes facts across data sources to the one time and location dimension as well as connects facts to KPIs. StreamCentral also auto-loads this database from various data sources into Big Data databases like HP Vertica or SQL Server
• Real-time event notification that can be consumed by operational applications via an API**
• Real-time event alerts
• Data marts that are automatically created, updated and managed by StreamCentral. The data marts denormalize data into a single table facilitating faster querying and analysis of data
• Real-time analytical data marts built that aggregates events and data across data sources to better understand conditions that influence events
• Real-time event data marts that bring together all relevant information for a single event• 360o data marts for association and pattern analysis**
** Coming Q3 2013
http://www.virtus-it.com 32
Sensors
Weather
Enterprise Applications
Data Visualization (Reporting, Analytics,
Dashboards)
Correlates Data
Generates Key Performance Indicators
Uncovers Events
Consumes real-time or static data OR Pulls data from
various data sources and
applies transformation and
standardization rules
Model Deployment Auto-builds database schema Auto-loads database Builds and continuous loads data to
event data marts Builds and continuous loads
Analysis Collections Publishes event data that can be
subscribed by Operational Applications
Devices
Auto-build Database Schema
360o Data marts and real-time data marts
Event Data Marts for every event along with its context as denormalized flat
tables
StreamCentralPush
Push
Massively Parallel Processing Systems - VerticaRDBMS – MS SQL Server
Publish event data to operational
applications – Web, mobile or desktop
StreamCentral Workbench – Big Data Solutions Modeler
Collate Raw Data (Push/Pull) – Real-Time or Static Model data standardization and transformation
rules Define business entities and connect raw data to
business entities Model Dimensions Model conditions to monitor across data sources Assign different conditions to different entities Model Correlation Rules Model events and specify context to add to events Model analytical data marts auto built by
StreamCentral
StreamCentral Big Data Server
Enterprise Applications
AP
I
TrafficA
PI
AP
I
API
http://www.virtus-it.com 33
builds two distinct types of analytical data marts
360o Data Marts** Real-Time Data Marts• Defined: Easily bring together and aggregate data
across data sources to get 360o insight. Analyze associations in data to determine patterns that impact business performance
• Define data mart structure by choosing the right set of attributes from data sources, KPIs, attributes from entities, and dimensions in the Workbench
• . StreamCentral auto-builds the data mart• Standardize data across time and location• Update data mart at pre-defined intervals
StreamCentral Data Marts are denormalized flat tables – Why?
• Defined: Aggregate real-time events and bring together data across data sources to analyze conditions that existed when events are detected
• Standardize data across time and location• Define data mart structure by choose the right set
of attributes from data sources, KPIs, events, attributes from entities, and dimensions . StreamCentral auto-builds the data mart
• Once data gets correlated in real-time data mart gets updated with appropriate insights
• Technology advancements in columnar data stores, bit map indexes, column indexes make it possible to scan and query large amounts of data in a single table
• Takes advantage of distributed architectures to scale out using commodity software• Supports :
• SQL Server columnar indexes• Vertica MPP
** Coming Q3 2013
http://www.virtus-it.com 34
StreamCentral Real-Time Operational Intelligence• Data Sources
• Import initial data load• Push data to StreamCentral API• Pull data from data sources at defined intervals • Apply transformations on the data in flight• SQL Server, Oracle, My SQL, REST API, SOAP Web Service, LDAP
• Auto connects data to time and location dimension
• Model entities. Connect data sources to entities
• Model measures and KPIs
• Model standard dimensions
• Model real-time correlation rule (to identify related records across data sources in real time)
• Model Events• Events based on real-time correlation rule• Event Data Mart (automatically gets created when event is detected)
• Requires real-time correlation• Brings together all data across data sources that were captured at the time the event was detected
• Model Real-Time Data Marts• Requires a real-time correlation rule• Update real-time data mart with streaming correlated data• Define attributes that make up the real-time data mart definition. Select subsets of information from : specific attributes from data sources, KPIs, events, entity
attributes, dimensions, time and location• Edit real-time data mart definition
http://www.virtus-it.com 35
StreamCentral 360o Data Aggregation**
• Data Sources• Import initial data load• Pull data from data sources at defined intervals • Apply transformations on the data• SQL Server, Oracle, My SQL, REST API, SOAP Web Service, LDAP
• Auto connects data to time and dimension location• Model entities. Connect data sources to entities• Model measures and KPIs • Model standard dimensions• Model 360o Data Marts
• Model 360 view query (define relationships across data sources to aggregate data)• Schedule batch update interval (typically hours)• Define attributes that make up the analysis collection. Select subsets of information from : specific
attributes from data sources, KPIs, entity attributes, dimensions, time and location• Edit and update data mart definition
• Define data level security** Coming Q3 2013
http://www.virtus-it.com 36
Data formats supported : • XML• JSON• String
Data Sources supported :• Database
• Microsoft SQL Server• Oracle• My SQL
• REST API• SOAP API• LDAP
• Specify transformation rules to data that is applied to data in flight
• Specify parameters when calling APIs• Auto fills location parameters based
on location data stored in the database about entities
• Auto creates tables in the backend database for data source data
Pull Data from Applications Push data to StreamCentral• StreamCentral REST API available to
stream data to StreamCentral – stream data from agents, sensors, probes, devices
• Specify transformation rules to data that are applied to the data in flight
• Auto creates the tables in the backend database for source data
StreamCentral Databases• Supports Microsoft SQL Server and
HP Vertica• Auto creates data structures in the
database for source data• Auto creates fact tables, dimensions,
flat tables for event analysis and flat tables for pattern and association analysis
• Data level security
StreamCentral Analytics• Device friendly visualization• Powerful portfolio of
visualization tools• Ability to embed in custom
applications• In-memory operations for fast
querying
StreamCentral Reports• Role based security• Subscribe to reports• Ability to embed in custom
applications• Export reports to various
formats
http://www.virtus-it.com 37
Transformation Description
LTRIM Removes all white spaces from the left
RTRIM Removes all white spaces from the right
Ignore Space Removes all white spaces from left, middle or right
Ignore Special Characters Returns string after ignoring all special characters
Contains Search for specific characters
Substring Extract a substring from a string
Left Removes the left part of a character string
Right Removes the right part of a character string
Replace Replaces specified string with another string
Startswith Search for a starting character
Endswith Search for an ending character
DoesNotContain Search for specific characters
Remove Remove specified characters or words from string
Range Search for a range
RoundOff Rounds off decimal value to a specific length
StreamCentral Transformations
• Easy to use transformations• Multiple transformations can
be executed on one attribute
http://www.virtus-it.com 38
StreamCentral Physical Architecture
http://www.virtus-it.com 39
StreamCentral Collector
Windows Server 2012, .Net Framework 4.5, MSMQ
StreamCentral Stream Processing Engine
Windows Server 2012, .Net Framework 4.5,, MSMQ
StreamCentral Stream Correlation Engine
Windows Server 2012, .Net Framework 4.5,, MSMQ
StreamCentral Data Engine
Windows Server 2012, .Net Framework 4.5,, MSMQ
All components can run on one machine
Every component can run on more than one machine
StreamCentral Cache Cluster
Windows Server 2012, .Net Framework 4.5, AppFabric
StreamCentral Metadata database
Windows Server 2012, Microsoft SQL Server2008 R2 or Microsoft SQL Server 2012
StreamCentral Database and data marts
Option 1:Windows Server 2012, Microsoft SQL Server2008 R2 or Microsoft SQL Server 2012 or SQL Server Parallel Data Warehouse
Option 2Linux, HP Vertica
StreamCentral Analytics
Windows Server 2012, Tableau Software
StreamCentral Physical Architecture and Software Requirements
http://www.virtus-it.com 40
1 server for StreamCentral Components:Collector, Stream Processing Engine, Correlation Engine, Data EngineCharacteristics of this server : Processor dependent therefore the higher the number of cores the better, medium cache and low disk storageSoftware: Windows Server 2012, .Net Framework 4.5, MSMQ
1 server for cacheHardware characteristics: : Cache dependent therefore more memory the better. Medium CPU and low disk storageSoftware : Windows Server 2012, .Net Framework 4.5, AppFabric
1 server for StreamCentral Meta Data Database, data mart storage and reportingHardware Characteristics:: High CPU, High Memory and High StorageSoftware : Windows Server 2012, SQL Server
1 server for StreamCentral Meta Data Database and reportingHardware Characteristics:: Medium CPU, Medium Memory and High StorageSoftware : Windows Server 2012, SQL Server
OR1 server for StreamCentral data martsHardware Characteristics:: High CPU, High Memory and High StorageSoftware : Linux, HP Vertica
+
StreamCentral suggested minimum system configuration
http://www.virtus-it.com 41
How does StreamCentral fit within your enterprise technology architecture?
http://www.virtus-it.com 42
Data Sources Method of Access
StreamCentral - Read data from Application
Application - Read data by subscribing to StreamCentral Real-Time Event API
Application - Read data by querying StreamCentral database
Enterprise Applications X real-time X
Using Web Service or REST API X real-time
Using database query X
Enterprise Service Bus X real-time X
via Web Service or REST API X real-time
via subscribing to messages X real-time
Enterprise Data Warehouse
via database query X X
Point databases via database query X X
LDAP via database query X
External Data Sources via Web Service or REST API X real-time
http://www.virtus-it.com 43
Sensors
Weather
Devices
Traffic
Custom Appl ica ti on sMai n fra me
Busi ness Servi ces
Enterpr i se Servi ce Bus - Messa gi n g / Med ia ti on / Orchestra ti on / Secu ri ty
Busi ness Process
Busi ness Process
Busi ness Process
Composi te Appl ica ti on
Composi te Appl ica ti on
Composi te Appl ica ti on
Auto-build Database Schema
Analysis Collections – Data marts as denormalized flat tables
Event Collections – Data Marts for every event along with its context as
denormalized flat tables
StreamCentral Engine
StreamCentral Workbench Collate Raw Data (Push/Pull) Standardize Data Define Business Rules Define Correlation Define events Define analytical data marts
auto built by StreamCentral
Historical Analysis
Real-time event data published to operational applications
and dashboards
Massively Parallel Processing Systems - VerticaColumnar databases with Bit Map indexes – ParStreamRDBMS – MS SQL Server
StreamCentral as part of an Enterprise Service Bus architecture
AP
I
ERP
Push
Pull
Push / Pull
http://www.virtus-it.com 44
Sensors
Weather
Devices
Traffic
ERP
Custom Appl ica ti on sMai n fra me
Busi ness Servi ces
Enterpr i se Servi ce Bus - Messa gi n g / Med ia ti on / Orchestra ti on / Secu ri ty
Busi ness Process
Busi ness Process
Busi ness Process
Composi te Appl ica ti on
Composi te Appl ica ti on
Composi te Appl ica ti on
Auto-build Database Schema
Analysis Collections – Data marts as denormalized flat tables
Event Collections – Data Marts for every event along with its context as
denormalized flat tables
StreamCentral Engine
StreamCentral Workbench Collate Raw Data (Push/Pull) Standardize Data Define Business Rules Define Correlation Define events Define analytical data marts
auto built by StreamCentral
Historical AnalysisEnterprise Business Intelligence
System
Massively Parallel Processing Systems - VerticaColumnar databases with Bit Map indexes – ParStreamRDBMS – MS SQL Server
StreamCentral and Enterprise BI as part of an Enterprise Service Bus architecture
Real-time event data published to operational applications
and dashboardsA
PI
Push
Pull
Push / Pull
http://www.virtus-it.com 45
Thank you for your time
Contact us for a demonstration
Stephen WellsCEO - Virtus IT LtdE: [email protected]: +44 77 111 30879
Raheel RetiwallaCTO - Virtus IT LtdE: [email protected]: +1 617 901 8370
A trusted partner