Think the Synchronization!

ari (masanari Motohasi) Institute of Mobile PC Technology

Purpose

● How many PC or PDA do you have?– Office, Home, Mobiles ... any environments...– We wish that I had access one Information anywhere.

● the complete networking– However, there are the barriers by cost, security, out of

range...● Do you have backup routinely?● We need to synchronize the information!

replication system

● high availability ( 高可用性 )– redundancy ( 冗長化 ) / ritualized

● consistency / concurrency ( 一貫性 ) – contention ( 衝突回避 )

● response time / reasonable ( 高性能 ) ● easy to use / manage● warm or cold standby

ultimately ubiquitous network● インターネットにどんなところからでもアクセス

– いつでも、どこでも、どんな方法でも

– 分散されていて、もっとも正規化が進む

– 情報の同期は考えなくていいのでは？

● しかし、ネットワーク的につながっている必要がある– セキュリティとプライバシは大丈夫？

– 同じセグメントかファイアウォールを抜く必要

– コスト、圏外、公衆電話？

Viewpoints

● like network layers (vertical)– physical/hardware to applications/software

● Transparency or not (horizontal).– normalized and distributed– “use caches” have the contended problem.– “make copies (backups)” have a lag time.– recovery time is very important

● clear the “contention problem”.● hot or cold standby?

Transparency

groupware / mail[communication]

mobile PC/PDA [any]

Version Controls[file]

backup [file](Tape etc)

SAN(snapshot)

Application/Software

Database[transaction]

Physical/Hardware[unit]

Removable MediaHDD/CF/SD..[file]

file sync. (rsync/ftpsync)remote access

inet (VPN / SSH)dial up [any]

web [page/service]

SAN / NAS /Internet Disk

grid/(mobile) agent[process/object]

RAID 1|5

Directionstelnet

remote display

dump/loaddatabase

P2P

Applications / Software

Web pages and services

● one of the most distributed and normalized solution with the Internet

● standard in the world– “standard” HTML / XML / SOAP etc– very reasonable

● with secure– SSL (server|client certification, encryption)– and the other – only good design (^_^;

Web pages – Just Try It!

● say nothing, because everybody knows (^ ^;)

Web Services – Just Try It!

● SOAP / XML on HTTP– Apache SOAP (Java), Microsoft SOAP toolkit, SOAP

module(Perl)– connect to the other world.

● Internet Applications (100% pure Java) and legacy systems (Microsoft COM (ASP / VB / C)).

● “Alive Check” system for an services (SOAP).● B2B applications (XML on HTTP POST)

SOAP(HTTP+VPN)J2EE

SOAPB2B

another sitesMicrosoft COM

Remote Console, Remote Display

● Remote Console– telnet (mainly UNIX)

● Remote Display / Desktop– VNC (Virtual Network Computing) *

● Windows and Unix– pcAnywhere *

● Windows only– X

● mainly UNIX– Terminal Server (Windows)

Remote Console, Just Try It

● telnet and telnetd (Unix)● Services for Unix (Windows) 29,800yen

– functions● telnet daemon (password sync Windows to Unix)● server/client/gateway for NFS● server for NIS

– for developers● all developers telnet, compile and test the applications

Remote Display, Just Try It

● VNC (Virtual Network Computing) – for developers and tests only– check the security

● pcAnywhere– control and administer to the servers on the branches

● for security patch and maintenance● terminal service (Windows 2000)

– another profiles

Groupware / Mail

● communication!– from mails to workflow systems

● Internet Mail System (on SMTP/POP3/IMAP etc)– very simple protocol– MIME

● Exchange / Notes– has replication options– share files, meeting agents, but depend on the special

platform

Internet Mail, Just Try It!

● Everybody knows in many OS and devices– Windows, Macintosh, UNIX, PDA, Phones, JRE...– devices and nodes

● for the synchronized files– send the attached mail oneself. (MIME)– it's not useful (not seamless and automatically)

● see “mailtrek” ...– however the firewall problem is clear! (with encrypt)

● check the contents and words

Groupware, Just Try It

● Microsoft Exchange– only Microsoft world( ?NET technology)

● Louts Notes / Domino– closed technology and culture (not open).– the Intranet and workflow systems

● have rich functions– mail system, scheduler, address books, public files,

databases, replications● depend on one company ... risk!

Database Management System

● to be one of most normalized– keep isolation by “transaction”

● SQL is the standard in the world– Oracle, Sybase ASE, Postage SQL, MS SQL...

● but mainly use in the LAN. ● 2 phase commit● replication server

– transaction base.

Database, Just Try It

● Sybase Adaptive Server Enterprise 11.0 – 12.5 *● Oracle8i *● Microsoft SQL Server *● My SQL● PostgreSQL

replication – mirroring

● write the many devices as same time

controller

WAN?

controller

read/write

applications / users

databases / devices / repositories(?)

replication – copy/ snapshot

● write the transaction to the primary– copy / snapshot to the secondary (only one way)

copydump

read / write

snapshot

replication - data trigger

● the trigger – one way– performance overhead– very complexly– rollback is not easy

read/write

! data updated

write

replications – stored & forward

differencedump transaction

queue, DB, file, transaction log

2 phase commit

● 2 phase commit protocol is not redundant– just distributed

are you ready?

yes yes NO

replications – bi-directions

read/write read/write

read only read only

site A site B

read/write

site A

read/write

site B

!contention

make the contentions

● when two or more “persons” update one information as same time.– objects(Beans), file, transaction, page, law...

update!

time

A

open

B

open

update

?

contention – lock

● blocking oriented– DBM, File Systems– use the wait queue– deadlock

time

A

open

B w

ait

update

update

openlock

B (lock waits)

contention - Optimistic Concurrency Control

● restart oriented– CVS, H/A EJB

time

A

open

B

update

update

re-open

openB

×

Version Controls

● not only “source” but “documents”● CVS [Concurrent Versions System]

– optimistic concurrency control method– shell (command prompt) and GUI– over HTTP, FTP, MAIL.– ported to any OS, but mainly in UNIX– controlling my documents.

● Microsoft Visual SourceSafe– for developers

File sync / replications

● rsync● Microsoft Products

– Briefcase (Windows 95 -) *– Replication Service (Windows NT)– Offline File Synchronization (Windows 2000 -)– Microsoft ActiveSync *

● Desktop PC – Pocket PC (serial / infrared)● another products

rsync, Just Try It!

● rsync on NT (run as services)– easy?

● rsync algorithm– low-bandwidth high-latency bi-directional

communications– great– maybe not secure, not use in the naked Internet...

Microsoft Briefcase, Just Try It

● Briefcase– ブリーフケースは直接ケーブル接続またはリムーバブル ディ

スクを使用してコンピュータ間のファイル転送を頻繁に行う場合に最適なツールです。ブリーフケースを使用して、メインコンピュータ上のファイルと他のコンピュータ上で変更したファイルを同期させることができます。複数のブリーフケースを作成してファイルを整理することができます

– easy to use, but performance is no good (many files with Gateway2000 Handbook).

Microsoft Offline Files● オフラインファイルはネットワーク上の共有ファイルを操

作するのに最適なツールです。オフライン ファイルを使用すると、ネットワークに接続していないときに変更した共有ファイルを次にネットワークに接続したときに同期させることができます。

● 使ったことがある方、どんな感じでしょう？

Microsoft Directory Replication Service

● “Windows NT Core Technology” で知りましたが、 実際の運用で使用したことはありません。– 使ったことがある方、どんな感じでしょう？

Microsoft ActiveSync, Just Try It

– Desktop PC – Pocket PC (serial / infrared)– Outlook data (mails, schedules, tasks, addresses...) and

files– easy to use

● but I am used to erased all data by miss operations.

ftptrek / mailtrek, Just Try It!

● ftptrek (Java)– synchronize with the data

● faster than Windows(SMB) copy● check the size and hash of the file● support the zipped, parallel and serial copies

– synchronize with database transaction– distribute over the program file updated

● mailtrek (Java)– SOAP on SMTP/POP3/IMAP? HTTP on MIME?– Sorry, under construction

Physical / Hardware

Remote Access

● very basic technology (PPP, SLIP...)– dial-up to the node (ex. office or home)– via Internet access

● the any services on PPP– wired and seamless!

Remote Access (Internet)

● same as dial-up, but NOT secure● SSH / VPN

– need: the secure server on the Internet– need: Global IP, Static IP or Dynamic DNS– need: keep the security and the privacy

● ex. Many site that has old version OpenSSH is attacked.● ex. Do dial-up users need to open the port 23(telnet) for the

Internet?– SoftEther Virtual Ethernet System

● ex. Ethernet on the HTTPS and SSH

VPN, Just Do It

● IPsec (commercial)– CISCO VPN 3005– connected to another site

● IPsec (private)– under construction– with FTTH (B-flets, NTT)– share the files, administrate the server

SoftEther Vitual Ethernet System

● SoftEther Virtual Ethernet System● Ethernet on the HTTPS and SSH● Able to pass through the firewall● Take care for use.

mobile PC / PDA● 皆さん、いろいろな使い方を教えてください！

● notebook PC● PDA

Storages

● SAN (Storage Area Networks)– fast/virtual Access (Fiber Channel)– snapshot make the backup in a second

● Network Attached Storages● Internet Disk

– use the Virtual Disk on the Internet Servers

SAN, Just Try It

● databases and file servers– critical missions

● snapshots is great– fast access (fiber channel and huge cache)– the backups and recoveries are in a second. – but the recoveries need the time

● full backup a day, and “loading transactions” are needed.

NAS (at home), try it

● HD-HLAN, Buffalo (Link Station)– change from the file server to HLAN (NAS)– low power, small size and silent– schedule backups the file a day via “rsync”

HLAN

no redundancy160GB

Godzillafile serverRAID 520GB

(4G×6)

Internet Hard Disk, Just Try It

● accessible to the disk like a drive● test the “MYDISK” service of rimnet.

– like the disk drive on the Windows● web accessible● 100MB, free● show demo...

– no accessible through our firewall● need the authentication to proxy

– stable?

Removable Devices

● HDD, Flash Memory(CF/SD)...– use as primary devices and backups– has the risks

● if lost (stop the job?)● if stolen (get out the information?)

● encryption– Encrypting File System (EFS) on windows 2000

● over NTFS 5.0– PGPdisk

RAID 1 or 5

● HDD is one of the most fragile parts– redundancy the HDD

Backup, Just Do It

● Tape– Just standard, but very expensive for personal user– serial access– DLT / DLT / LTO...

● HDD– Random access– I use the bus-powered 2.5inch HDD with rsync (at

home)

Enterprise Disaster Recover Site,Just Do It

● for Business Continuity Planning (BCP)– Reduce the risk! plug off?

● redundant the anything– Database synchronized by ftptrek– Wires, networks (include Internet) and servers– humans and business processes

● warm standby– if “the incident” is happened, switch in a minute.

Thank you!