Takeo Higuchi (KEK); CHEP.20120521.pptx

High Speed Data Receiver Cardfor Future Upgrade of Belle II DAQ

1. Introduction – Belle II DAQ • Experimental apparatus parameters

SuperKEKB accelerator Maximum luminosity 8x10³⁵ /cm²s

Belle II detectorExpected raw data size 1MB + 80kBMaximum L1-trigger rate 30 kHzDead-time fraction < 3%

The anticipated faint signatures of the new physics effect are detected with a large number of e⁺e⁻ collisions and a dedicatedly configured hermetic and granular detector.

• Belle II detector and Belle II DAQ

Near-detectorsignal digitizersand optical TX

Opt TX

Far-site optical RX ~700 RocketIO

optical links

COPPER readout system

COPPER boards (x~180)

~20MB/sper link at most

HSLB cards (x~700)• x4 RX-card slots.• Pipeline buffer (4MB).• PrPMC slot for online processing.• Local and PCI buses bridged by

PLX9054 bus-bridge chip.• x2 GbE data-output ports.

EVB (PC)

HLT~⅓ rate

reduction

Storage~1.5GB/s

•Max link speed =3Gbps.•Virtex-5 FPGA.

2. New Data Receiving System

PXD 2-layer DEPFET pixel detectorSVD 4-layer silicon vertex detectorCDC Central drift chamberTOP Time of propagation counters

ARICH Ring-image Cherenkov countersECL Electromagnetic calorimetersKLM KL and μ detector

TRG Hardware trigger information

Opt RX

T. Higuchi et al.,IEEE Trans. Nucl. Sci. 52, 1912 (2005).

~20m fibers

• Design concept of the COPPER board

~

Opt RX.

Opt RX.

Opt RX.

Opt RX.

.PrPMCFIFO

FIFO

FIFO

FIFO

9054

Local bus PCI bus

COPPER board (VME9U size)

• A new data receiver card

DDR3memory(up to 4GB)

Virtex-6 (XC6VLX75T)

Statusindicators (LED)

x2 optical inputports (SFP+)

x2 LVDS outputports (RJ45)

PCIe x4Gen2

New data receiver card (PCI Express) • Optical RX → The receiver card itself • COPPER FIFO → On-card DDR3 • Local/PCI buses → PC’s motherboard bus • PrPMC → PC’s CPU • COPPER GbE → PC’s GbE port

Design concept of the new receiver card

Use of commodities saves costs for software R&D and efforts for system maintenance.

DDR3 SFP+#1 SFP+#2

PCIex4 Gen2

Virtex-6

Mem

ory

I/F

Gen

erat

or

FIFO

(1k

x 32

8b)

FIFO

(1k

x 25

6b)

FIFO

(1k

x 4b

)

AURO

RA8b

10b

AURO

RA8b

10b

FIFO

(1k

x 28

8b)

FIFO

(1k

x 28

8b)

AURO

RAco

ntro

ller

FIFO

(1k

x 28

8b)

FIFO

(1k

x 28

8b)

AURO

RAco

ntro

ller

DDR3 controller

DM

Aco

ntro

ller

PCIe

cont

rolle

r

Inte

grat

edbl

ock

LVDS outputs to notify the DDR3 getting full.

3. Performance Studies• Prototype study

Virtex-6 evaluationcard (ML605)

FMC

Mezzanine cardwith x4 SFP+

Loop back

• New card’s performance study

Loop

bac

k

New datareceiver card

DMA

PCIe x4 Gen2

Readout PCCorei7 3.4GHzx2 (PCIe x8 Gen3)4GB main memoryScientific Linux 6.0

•Optical link performance †

BER < 1.8x10⁻¹³•DMA performance †

BER < 1.9x10⁻¹³Throughput = 635MB/s † 4MB per transaction

• The COPPER board can handle >30 kHz L1 trigger rate with a 90% dead-time reduction than Belle I proved in the Belle I DAQ operation.

• The COPPER system is widely adopted in many HEP experiments.

The great success of the COPPER system:

L1-asynchronous DAQ by FIFO Online data reduction by CPU

Disadvantage of the COPPER system:• As the COPPER system embeds everything needed for the DAQ, the

system became too complicated for the average Belle II users to start up and to maintain.

Simple-minded data receiver More use of commodities

Block diagramof the new data

receiver card

.GbE

Stud

ies

in c

oope

ratio

n w

ith S

anse

i Sys

tem

Inc.

DMA

PCIe

x8

Gen

1

DELL 760MTCore2 Duo 3.0GHzPCIe x164GB main memoryCentOS 5.5

16kB image pre-stored in ML605

Opt TX8b10b

8b10bOpt RX PC

Loop back DMA/PCIe

•Optical link +DMA performance †

Throughput = 470MB/s

† Average over8192 transactions

We continue system R&D with a milestone settled in the middle of 2017, when the peak luminosity will exceed 2x10³⁵ /cm²s.

Takeo Higuchi,Nobu Katayama,

High Energy Accelerator Research Organization (KEK) Kavli IPMU, University of Tokyo