Icse2008 enterprise integration_with_soa_broecker

Seminar Integrationsarchitekturen

© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 1

Enterprise Integration with SOA

Christoph Bröcker

Half Day Tutorial, ICSE 2008, Leipzig


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 2

Objective of the Tutorial

Answer questions on Enterprise Integration and its relation to Service-Oriented Architecture

• Why is Enterprise Integration (EI) so hard?

• How can the typical activities around EI be structured?

• EI, EAI, SOA and BPM - how are all these buzzwords related?

• Is SOA the single answer to the questions of EI?

• What is flexibility and how can we achieve it?


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 3

Agenda

14:00 – Welcome

14:15 – Defining Enterprise Integration (EI)

14:30 – Why is EI so hard?

15:00 – Addressing EI with SOA

15:30 – Coffee Break

16:00 – Beyond SOA: Integration Patterns

16:30 – Structuring EI Activities

16:45 – Discussion: Flexibility

17:15 – Summary

17:30 – End


© Zühlke 2008

18. Februar 2008

Dr. Klaus Alfert Dr. Christoph Bröcker

Slide 4

Defining Enterprise Integration


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 5

Enterprise Integration

Definition:

The practice of ensuring efficient execution of business processes across organisational and technical boundaries within an enterprise and its business partners.


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 6

CRM Research Information Market data

Investment Banking

Payment Settlement

Credit Credit Products

Money market / Foreign

Currencies

Shares Derivatives (incl. OTC) Bonds

Interest and Credit

OTC- Derivatives

Subsidized Loans

Clearing Clearing

Risk mgmt/ Collateral mgmt.

Execution Position management/ Rating

Order/ Execution

Assessment/ Decision

Acquisition

Payment Settlement

Payment Creation

Risk mgmt/ Collateral mgmt.

Pricing

Windows

Mainframe

Unix

Other

Example: IT system landscape of a bank


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 7

Layers of Enterprise Integration

Business Integration

Strategy Organisation

Processes

IT Systems

Enterprise System Integration

Infrastructure

Enterprise Integration


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 8

Enterprise Integration = EAI?

This depends on your interpretation of EAI (Enterprise Application Integration).

The academic/conceptual view

• EAI is about integrating applications within enterprises

• EAI is close to EI, but often with less focus on processes and business integration

The commercial/product view

• EAI was a trend / buzz word of the 90’s

• EAI products are proprietary and centralistic

• EAI is not state-of-the-art anymore


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 9

Enterprise Integration and Diversity

Enterprise Integration is about Alignment:

• Different systems

• Different goals

• Different owners

• Different roadmaps

Enterprise Integration is NOT:

• Integration within one (large) application


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 10

Why Enterprise Integration matters

Internal and external customer demands

• Single customer view

• Seamless processes across channels

Stronger market pressure

• Globalisation

• Regulation

Typically, companies spend 80% or more of their IT budgets just to maintain existing systems.”

“Fighting Complexity In IT”, The McKinsey Quarterly, 2003

Growing Expectations

Increasing “Legacy”

More existing systems

More processes are (partially) automated


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 11

Business Impact of Enterprise Integration

Time

• How fast can you react to market impulses?

• Are your best people available for key projects?

Cost

• Can you keep your pricing competitive?

• Is your procurement flexible?

Risk

• Do you understand your system dependencies?

• Are you dependent on your infrastructure vendor?


© Zühlke 2008

18. Februar 2008


Slide 12

Why is Enterprise Integration so hard?


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 13

ABC Service Platform

Application Services

Operation & Maintenance

Network Elements

Common Services

User Interface Admin Interface

Network Services Provisioning

Network Listener

Partner Staff Value Added Service

Systems Management

Alarms

Operator Staff

Example: Architect‟s view (Planning)


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 14

Data

Routing

(DR)

LDAP-DB

JMS / ABU

Internet

Internet

Internet

distributed

Proxy

distributed

Proxy

RAS / GGSN

RAS / GGSNAMBC

OTA Server

FSP

TCS

Mediation

Micropayment

Administration

Interface

Portal

GMGC

IMS

BM

User

Repository

corporate

DB

prepaid

subscriber provisioning /

subscriber status changes

postpaid

Push

SOAP (XML)

HTTP

WEB

WEB

WEB

WEB

LDAP

replica

LDAP

for:

- Subscribers

- 3rd Parties

- Marketing

- Administrators

- Customer Care

Billing / Logging

WAP

LDAP

Billing / Logging

Management (O&M)

WEB

WAPRadius

accounting

Radius

accounting

WAP

WEB

CMS

Proxy / Gateway(central component)

Billing / Logging

RADIUS

LDAP

replica

PAP

LISA

Example: Administrator‟s view (Operating)


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 15

Divergence as the Adversary of EI

Consistency

Order

Simplicity

Complexity

Confusion Unpredictability

Irregularity

Chaos

Systems have a natural tendency to erode over time.

Understanding Control

•New/changed requirements

•Cost/time constraints

•Technical issues

•Unexpected scaling

•Radical business / technology shifts

•Mergers & acquisitions

•COTS purchases


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 16

Challenges of Enterprise Integration

Awareness

Care

Structure

Needs

Change


Organizing the issues, so that they can be addressed.

Source: Zühlke Applied Framework for Integration and Reuse (ZAFIR)


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 17

Awareness

EI Issue No. 1 Most enterprises have no systematic approach towards system convergence and EI.

• To stay effective, enterprises have to act locally – Local responsibility and budgets – Local incentives and KPIs – Local problems and solutions

• Systems are built for immediate business purpose, ignoring mid-term integration needs

• Organizations need to create an awareness of the forces of divergence – Coordination and alignment come at a cost

Awareness

Care

Structure

Needs

Change



© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 18

Needs

EI Issue No. 2 There is no one-size-fits-all integration strategy.

• Organizations are not alike – Size and structure – Degree of centralization in company culture – Speed of market dynamics – Automation of supplier/customer interaction – Frequency of mergers and acquisitions

• Organization requirements differ – Breadth of technology used – Human interaction vs. Straight-Through-Processing

• Non-functional requirements matter – Availability, reliability, response times, security,

operability, maintainability, …

Awareness

Care

Structure

Needs

Change



© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 19

Structure

EI Issue No. 3 There are countless choices for linking systems.

• Design choices – Loose vs. tight coupling – Asynchronous vs. synchronous communication – Central vs. decentral deployment

• Technical choices – Communication protocols and data formats – Standards (e.g. WS, CORBA, EDIFACT) – Products

• Vendors offer limited help – Integration products are complex by nature – Guidelines for product usage are essential – Enterprises need “loosely coupled” integration components – Vendors have an interest in “tightly coupled” integration suites

Awareness

Care

Structure

Needs

Change



© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 20

Change

EI Issue No. 4 Integration projects are hard to manage.

• Coordination of people, not just systems – Large set of stakeholders – Automation requires agreement on processes – Analysis requires much time – No formal authority over other teams

• Lack of proven integration delivery methods – Documented software engineering processes focus on

building rather than integrating – Integration products often do not blend into the tool chain – Testing integration solutions is difficult

Awareness

Care

Structure

Needs

Change



© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 21

Care

EI Issue No. 5 Desired integration structures cannot be fully realised and keep changing.

• The investment in existing systems and integration technology is huge – Desired structure cannot be built in a big bang – Daily business continues and budgets are limited

• Systems have longer lifetimes than technology. – This includes integration technology – Integration infrastructure becomes legacy itself

• Standards have to “live” to stay relevant (e.g. CORBA)

Awareness

Care

Structure

Needs

Change



© Zühlke 2008

18. Februar 2008


Slide 22

Addressing Enterprise Integration with SOA


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 23

Awareness: SOA as an opportunity for IT teams

SOA has management attention, which can be leveraged

• Create awareness of divergence

• Define integration strategy

• Improve existing technology and practices

Danger of viewing/selling SOA as a “quick fix”

In most organisations, SOA is just another IT topic

+ –


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 24

Needs: SOA is a style that needs tailoring

SOA is just a set of concepts

• Organisations can choose those that fit their needs

• Large degree of freedom in SOA realisation

SOA addresses typical needs of many organisations

• Heterogeneity

• Reuse of existing systems

SOA is just a set of concepts

• SOA is not a finished integration strategy

• Many practical problems remain

Danger of ignoring real needs

• Reduction of SOA to a standard (web services) or a product

+ –


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 25

Structure: Does SOA just add to the mix?

SOA + – Service composition is an important integration pattern

Basic web services have wide support across the industry

Adoption of SOA principles leads to systems that are designed for integration

Not everything is best modelled as a service

Immature/overrated standards

• QoS-enhanced web services (WS-*)

• UDDI, BPEL, JBI

Uniformity of SOA

• Not all problems require orchestration

Abstraction can lead to inefficiency

WS Base Notification

WS Addressing

WS Trust

WS Secure Conversation

WS Security

WS Policy

WSDL SOAP

WS Topics

WS Brokered Notification

WS RM WS

Coordination

WS Atomic Trans.


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 26

Change: Projects are still hard within a SOA

+ – SOA provides help to integration projects in some areas

• Service modelling

• Capturing process requirements (e.g. BPMN)

Many aspects of executing projects against a SOA remain unclear

• Managing stakeholders

• Rewarding creation of reusable services

Benefits of SOA are mostly mid-term and not for the project

• Problem of motivating projects to adopt SOA


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 27

Care: The realm of SOA governance

+ – Discussions of SOA Governance often capture the important aspects of system evolution over time

SOA Governance as an opportunity for more alignment/convergence

Governance is sometimes defined too narrow

• Cataloguing services

• Enforcing reuse

Danger of isolation

• SOA governance must be linked with existing alignment processes (e.g. project portfolio management)


© Zühlke 2008

18. Februar 2008


Slide 28

Beyond SOA: Integration Patterns


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 29

Integration Patterns as a specific vocabulary

The Building Blocks: Coupling Patterns

Technical

Logical

Coordination

File Transfer Shared DB Messaging RPC

Screen Scraping API

Data Resources Services Events

Point-to-Point Mediation Coordination

Choreography Orchestration Workflow

DB Replication


© Zühlke 2008

18. Februar 2008


Slide 30

Direct Data Integration

Coupling patterns

• File Transfer / Shared DB

• Data Orientation

• Point-to-Point

Often called „Legacy“

Many dependencies

Fear of changing anything

Data

Routing

(DR)

LDAP-DB

JMS / ABU

Internet

Internet

Internet

distributed

Proxy

distributed

Proxy

RAS / GGSN

RAS / GGSNAMBC

OTA Server

FSP

TCS

Mediation

Micropayment

Administration

Interface

Portal

GMGC

IMS

BM

User

Repository

corporate

DB

prepaid

subscriber provisioning /

subscriber status changes

postpaid

Push

SOAP (XML)

HTTP

WEB

WEB

WEB

WEB

LDAP

replica

LDAP

for:

- Subscribers

- 3rd Parties

- Marketing

- Administrators

- Customer Care

Billing / Logging

WAP

LDAP

Billing / Logging

Management (O&M)

WEB

WAPRadius

accounting

Radius

accounting

WAP

WEB

CMS

Proxy / Gateway(central component)

Billing / Logging

RADIUS

LDAP

replica

PAP

LISA


© Zühlke 2008

18. Februar 2008


Slide 31

Integration Broker

Coupling patterns

• File Transfer / Messaging

• Data, Services, Events

• Mediation

Central Engine

• Mapping, Protocol transformation, Routing

Example: EAI platforms from the 90„s

• Some „ESB“ products are still centralised

System D System C

System A System B

System E System F

Broker


© Zühlke 2008

18. Februar 2008


Slide 32

Integration Bus

Coupling patterns same as Integration Broker

Differences from Integration Broker

• Flexible deployment to different machines

• Lightweight mediators – Example: Adapter – Deployment nearer to the source systems

Example: Some ESB products

System D System C

System A System B

System E System F

Bus

Bus

Bus


© Zühlke 2008

18. Februar 2008


Slide 33

Coupling patterns

• RPC, (Messaging)

• Services

• Point-to-Point, Orchestration

Services are coarse-grained and business-relevant.

Basis services form the foundation of a hierarchy of enhanced services.

Anti-Pattern: JBOWS

Service Composition

Process-centric services

Basic services

Applications / Databases

Enhanced services


© Zühlke 2008

18. Februar 2008


Slide 34

Process Integration

Coupling patterns

• Any technical and logical coupling

• Coordination: Workflow

Manual activities and IT applications joined through a common process.

• Planning

• Execution

• Optimisation

Business Process

Automated activities

Applications / Databases

Manual activities


© Zühlke 2008

18. Februar 2008


Slide 35

User Interface Integration

Coupling patterns

• RPC, Shared DB, Screen Scraping

• Resources, Services

• Point-to-Point

„Superficial“ Integration

• Limited support for complex operations

• Often only a „quick win“ add-on for a CMS

Benutzerschnittstellen einzelner

Anwendungen

User interfaces of single

applications

Portals Workflow systems


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 36

SOA as a series of integration patterns

SOA = Integration Bus (based on Services)

+ Service Composition

+ Process Integration BPM

+ Integration Bus (based on Events) EDA

Some people use the term SOA in a broader sense

• Every interface is a “service” (data, events, resources, messages, DB lookups)

Optional


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 37

Real-world example: Integration architecture of a bank

Four complementary enterprise integration domains

Best-of-breed product suite

• Integrating the integration products

EAI

SOA

BPM

User Interaction

Integration architecture as a combination of patterns


© Zühlke 2008

18. Februar 2008


Slide 38

Structuring Enterprise Integration Activities


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 39

From integration of systems...

Business Processes

Domains

Systems

Enterprise Partner


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 40

...towards integration of people

Business Processes

Domains

Systems

Enterprise Partner

CEO CIO

Project manager

Process Manager

Domain Architect

Operations Manager DB

Admin Support Engineer

Business Partner

External Management

System Architect A

Development Team A

System Architect B

Development Team B

Outsourced Admin

SLA Manager

User

Department Head

Team Lead External

Architects External Consultants

External Developer


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 41

Real-world example: Integration activities at a telecommunications provider

Lawful interception (TKG, TKÜV)

Consolidation of billing systems

Service portal R3 Service portal R4

Introducing TIBCO iProcess v10.5

Time

Upgrade v11.0

Reporting improvements

New mobile flat rates

Triple play contracts Fixed/mobile/DSL

Coordination (e.g. SOA Governance / EAM / PPM)

Mix of business and infrastructure

projects


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 42

Iterative Integration: Strategy and Tactics

Approach Cycle

Projects/Solutions Tactics

Integration management, EAM, Governance Strategy

Run Solution

Build Solution Assess Influencers

Define Initiative

Assess Influencers Define Integration Architecture

Define Approach

Govern Architecture

Alignment

Build Solution

Define Initiative

Govern Architecture

Initiative Cycle


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 43

Integration Architecture: A moving target

EI is a process, not a goal

Target Integration

Architecture

Actual Integration

Architecture

2008 Q2 2009

2008 Q3

2008 Q4

2009 Q1

2010


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 44

Convergence ≠ Central EAI Platform

Convergence Examples

• Bringing people together – Incentives/triggers for physical meetings – Shared concepts – Common goals

• Enterprise (integration) architecture management – Central coordination – Strong local control

• Common data models – Glossaries – Agreed data structures

• Harmonized processes – Incorporating architecture into the project process – Project post mortems – Shared metrics


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 46

Quotes on flexibility

“The move towards a SOA is a key element of software flexibility.”

“Achieving IT Flexibility”, Butler Group, 2006

“Increase flexibility in your SOA when you apply the seven levels of maturity on the path to SOA adoption.”

The Service Integration Maturity Model (SIMM) IBM developerworks, 2005

“By deploying Enterprise Service-Oriented Architecture (Enterprise SOA), you can increase flexibility and control your costs better.”

“Flexibility with Enterprise SOA” SAP SDN, 2006

“TIBCO iProcess Decisions … [provides] the flexibility and agility to respond to a constantly changing business environment. ”

Decision Datasheet TIBCO, 2005

„IT service management secures IT flexibility“ Computer Zeitung, 2. Mai 2008


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 47

Definition

Flexibility (in engineering):

… is the ability of a system

• to respond to potential internal or external changes affecting its value delivery

• in a timely and cost-effective manner

Flexibility is only required and meaningful in the presence of uncertainty.

Source: Wikipedia


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 49

Flexibility as a Bet

Characteristics of Bets

• You invest a certain amount of money

• If the future develops as planned, you win

• If not, you lose

Examples of Bets in IT

• Enabling specific changes – Abstraction layers – Declarative configuration of certain parts – Business rules – Exposing services

• Enabling general change – Adoption of main-stream technology – Adherence to standards (e.g. Web Services, XML)


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 50

Source: W. Baermann, Wikimedia Commons, Lizenz CC Attribution-Share Alike 2.5 Generic


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 51

Flexibility through Hygiene

Characteristics of Hygiene

• Small investments at regular intervals retain value

• If omitted occasionally, impact is negligible

• If omitted continually, impact is severe

Examples of Hygiene for IT systems

• Supporting consistency – IT strategy, standards and guidelines – Architecture reviews

• Maintaining simplicity – System consolidation – Refactoring – Know-how exchange – Infrastructure projects


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 53


Enterprise Integration matters

• Integration is a key issue of enterprises today

• The challenges of EI are here to stay

• EI is a discipline, not a solution

SOA can support EI in many ways

• Conceptually, technically, organisationally

SOA is not the only answer to the challenges of EI

• SOA can be viewed as a combination of integration patterns, but there are others

• A more precise vocabulary is needed

• When will the next buzz word arrive?


© Zühlke 2008

18. Februar 2008


Slide 55

Further reading

• Bloomberg, Schmelzer: Service Orient or Be Doomed, Wiley 2006

• Chappell, Enterprise Service Bus, O‘Reilly 2004 • Conrad et al: Enterprise Application Integration,

Spektrum Verlag 2006 (in German) • Gold-Bernstein, Ruh: Enterprise Integration,

Addison-Wesley 2005 • Hohpe, Woolf: Enterprise Integration Patterns,

Addison-Wesley, 2004 • Krafzig et al.: Enterprise SOA, Prentice-Hall 2004 • Schelp, Winter (ed.): Integrationsmanagement,

Springer 2006 (in German) • Trowbridge et al: Integration Patterns, MS Press 2004


© Zühlke 2008

12 May 2008

Christoph Bröcker

Slide 56

Contact details

Christoph Bröcker

Business Unit Manager Zühlke Engineering GmbH Düsseldorfer Str. 40a 65760 Frankfurt (Eschborn)

[email protected]

Your feedback is welcome!

Icse2008 enterprise integration_with_soa_broecker

Technology

Transcript of Icse2008 enterprise integration_with_soa_broecker