The Operating System of the NeuromorphicBrainScaleS-1 System

Eric Müller∗‖, Sebastian Schmitt∗‖, Christian Mauch∗‖,Sebastian Billaudelle‖, Andreas Grübl‖, Maurice Güttler‖, Dan Husmann‖, Joscha Ilmberger‖, Sebastian Jeltsch‖,

Jakob Kaiser‖, Johann Klähn‖, Mitja Kleider‖, Christoph Koke‖, José Montes‖, Paul Müller‖,Johannes Partzsch§, Felix Passenberg‖, Hartmut Schmidt‖, Bernhard Vogginger§, Jonas Weidner‖,

Christian Mayr§ and Johannes Schemmel‖

∗contributed equally‖ Kirchho�-Institute for Physics

Ruprecht-Karls-Universität Heidelberg, GermanyEmail: {mueller,sschmitt,cmauch}@kip.uni-heidelberg.de

§ Chair of Highly-Parallel VLSI-Systems and Neuro-MicroelectronicsTechnische Universität Dresden, Germany

Abstract—BrainScaleS-1 is a wafer-scale mixed-signal ac-celerated neuromorphic system targeted for research inthe �elds of computational neuroscience and beyond-von-Neumann computing. The BrainScaleS Operating System(BrainScaleS OS) is a software stack giving users the possibil-ity to emulate networks described in the high-level networkdescription language PyNN with minimal knowledge of thesystem. At the same time, expert usage is facilitated byallowing to hook into the system at any depth of the stack.We present operation and development methodologies im-plemented for the BrainScaleS-1 neuromorphic architectureand walk through the individual components of BrainScaleSOS constituting the software stack for BrainScaleS-1 plat-form operation.

I. IntroductionState-of-the-art neuromorphic architectures pose many

requirements in terms of system control, data preprocessing,data exchange and data analysis. In all these areas, softwareis involved in satisfying these requirements. Several neuro-morphic systems are directly used by individual researchersin collaborations, e.g., [1–4]. In addition, some systemsare operated as experiment platforms providing access forexternal users [1–3, 5].

Especially systems open for a broader range of usersrequire clear and concise interfaces. Neuromorphic platformoperators have additional requirements in resource manage-ment, runtime control and —depending on data volumes—“grid-computing”-like data processing capabilities. At thesame time, usability and experiment reproducibility arecrucial properties of all experiment platforms, includingneuromorphic systems.

Modern software engineering techniques such as codereview, continuous integration as well as continuous de-ployment can help to increase platform robustness andensure experiment reproducibility. Long-term hardwaredevelopment roadmaps and experiment collaborations draw

attention to platform sustainability. Technical decisions needto be evaluated for potential future impact; containing andreducing technical debt is a key objective during planningas well as development. Regardless of being software-drivensimulations/emulations, or being physical experiments, mod-ern experiment setups more and more depend on theseadditional tools and skills in order to enable reproducible,correct and successful scienti�c research.

This paper describes the results of a ten-year projectdelivering the software environment and platform operationtools for the BrainScaleS-1 neuromorphic system. Thefollowing sections describe the hardware substrate and givea general overview. Section II introduces the methods andsoftware tools we employ. In section III, the scopes andimplementation details of the main software layers andlibraries are explained, followed by an overview over theoperation of the platform in section IV. Section V exempli�esthe usage of the BrainScaleS Operating System on a simpleexperiment and describes larger experiments carried out inthe past. We close in section VI with an overview over futuredevelopments and discuss our endeavor and the lessonslearned in section VII.

A. The BrainScaleS-1 Neuromorphic System

Classical neuromorphic systems make use of VLSI to imple-ment electronic analog circuits mimicking neuro-biologicalarchitectures in the nervous system [6]. Contemporarysystems also employ mixed-signal techniques to enable�exible system connectivity based on conventional digitalinterfaces [7]. Recently, purely digital systems emerged [1,3, 4]. Compared to the analog approach typical advantagesof such systems are: deterministic behavior and arbitrarilyprogrammable neuron dynamics. However, when comparing

arX

iv:2

003.

1374

9v1

[cs

.NE

] 3

0 M

ar 2

020

F

GEI

CA B

G

D

H

HH

(a) (b)

Fig. 1: (a) 3D-schematic of a BrainScaleS Wafer Module (dimensions:50 cm × 50 cm × 15 cm) hosting the wafer (A) and 48 FPGAs (B).The positioning mask (C) is used to align elastomeric connectorsthat link the wafer to the large main PCB (D). Support PCBs providepower supply (E & F) for the on-wafer circuits as well as access (G)to analog dynamic variables such as neuron membrane voltages.The connectors for inter-wafer and o�-wafer/host connectivity(48 × Gigabit-Ethernet) are distributed over all four edges (H) ofthe main PCB. Mechanical stability is provided by an aluminumframe (I). (b) Photograph of a fully assembled wafer module.

at the same technology node the energy as well as areaconsumption is increased.

Based on the ideas of a single-chip implementation calledSpikey [8], BSS-1 is a mixed-signal architecture providingaccelerated Adaptive Exponential Integrate-and-Fire (AdEx)neuron dynamics and plastic synapses [5, 9, 10]. While manyneuromorphic systems target biological real-time execution(i.e. model time constants in the same order as their biologicalcounterpart) [11], BSS-1 evolves in continuous time, typicallyat a speed-up factor of 1000–10 000 faster than biological realtime. Consequently, real-time interfacing to, e.g., sensors androbotic applications are not the main goal of the architecture.The design focuses on fast model dynamics, controllablemodel parameters and system scalability, thereby allowingfor time-compressed emulations of longer experiment timescales. Plasticity and learning processes can therefore beinvestigated in manageable time frames.

Figure 1 depicts a BSS-1 wafer module. The main con-stituent is a silicon wafer, manufactured in 180 nm Comple-mentary Metal Oxide Semiconductor (CMOS) technology,carrying 384 HICANN chips that are interconnected via anon-wafer bus network. Each chip hosts up to 512 AdExneurons and 113k plastic synapses. 48 Xilinx Kintex-7Field Programmable Gate Arrays (FPGA) provide an I/Ointerface for con�guration, stimulus and recorded data. Theconnection between these FPGAs and the control clusternetwork is established via 1-Gigabit and 10-Gigabit Ethernet,cf. Schmitt et al. [12].

B. Performing Experiments

Providing access to and experimenting with neuromorphicsystems is an active �eld of research [13–16]. In 2017 Schu-man et al. [17] highlighted that:

Supporting software will be a vital component inorder for neuromorphic systems to be truly successfuland accepted both within and outside the computingcommunity.

We second this statement and this paper details our approachon how to tackle the software challenge in the contextof BSS-1 in particular. However, many of the developedsolutions are of general scope and should be applicable toother systems that have similar features and targets.

Di�erent approaches to user interfaces are viable, e.g.,the interface for Spikey —a previous chip developed by theHeidelberg group— mainly used Python for con�gurationand experiment description [18]. In contrast, BrainScaleSOS provides only thin Python wrappers for all user-facingApplication Programming Interfaces (API) while the coresoftware is written in C++, cf. section II-B2. Spikey focusedon PyNN as the experiment description language [19], seesection II-B1. As of today, common spiking neural networksimulators and a few hardware emulators support PyNN [13,19]. Further work investigated the typical work�ow whenporting experiments from pure software simulations toneuromorphic platforms [20].

BSS-1 has a fairly large parameter space, O(50 MiB perwafer) of static con�guration data, and the analog char-acteristics of the system require expert knowledge whencon�guring the system on a low abstraction level. Thus, aneasy-to-use interface and the support of non-expert usersare the main objectives of the BSS-1 software developmente�ort. In addition to the neuroscienti�c API, the currentBSS-1 software stack provides access a multitude of inter-faces to manipulate the experiment description and systemcon�guration on lower level, cf. sections III-A and III-B. Forexample, con�guring certain hardware entities in a manualfashion but still being able to rely on the automated processin other areas facilitates the commissioning of new systems.This approach allows for exposing expert-level manipulationof the complex neuromorphic substrate without giving up allbene�ts of automation. Finally, increasing numbers of plat-form users, the parallel operation of production-type systemsas well as commissioning future hardware generations posechallenges for the development and operation methodologies:platform robustness, experiment reproducibility and, to alesser extent, the reduction of turnaround times betweenhardware revisions are essential.

C. Experiment Platform

Providing the research community with access to neuro-morphic systems has been an ongoing e�ort pushed by large-scale research projects, such as the Human Brain Project.In the case at hand, a large-scale neuromorphic system isoperated in a multi-user setting. From a high-performancecomputing (HPC) perspective, “fast” neuromorphic systemsresemble spiking neural network accelerators. To e�cientlyutilize the available hardware systems requires solutions thatare also common in HPC centers: resource management,time-sharing, fairness, accounting, monitoring and visual-ization. However, providing access to external researchers

also increases the need for robust operation, experimentreproducibility and support.

II. Methods and ToolsPerforming well-controlled experiments on the BSS-1

system is the main task of BrainScaleS OS. In broad terms,experiments are de�ned by what and when, representingthe data and the control �ow. APIs for spiking neuralnetwork (SNN) descriptions, e.g., PyNEST or simulator-agnostic PyNN typically focus on the initial experimentsetup, i.e. network topology, model parameters, plasticityrules, recording settings and the stimulus de�nition. Similarly,neuromorphic hardware requires an initial con�guration thatis typically performed before any stimulus is connected tothe SNN.

Neuromorphic hardware is also di�erent, as it oftenneeds additional —more technical— settings compared tosimulators which numerically calculate the time evolutionof di�erential equations as, e.g., described by Einevoll et al.[21]. In our case, APIs that solely concentrate on aspectsof neuron and synapse models, network topology andstimulus are insu�cient, especially during commissioning.Usability for both experts and non-experts is the key featureof the software stack. The main points of usability are:1) encapsulation of domain knowledge into software layers;2) validity checking of settable parameters; 3) error reportingand explicit error handling; 4) consistency in the API layersconcepts, and their representation of hardware entities;5) availability of tested settings and con�guration protocolsto the user; 6) possibility to inject customized behavior atall levels of the software stack.

The development of the BrainScaleS-1 platform startedalready in 2008. Though many hardware and software compo-nents are now over a decade old, updates and improvementsare continuously being made. In this section, we shortlydescribe the development methodology and foundations forthe BrainScaleS OS.

A. MethodologyCompared to previous e�orts made when developing the

software stack for Spikey [18], large-scale neuromorphicsystems introduce additional complexity. For example, multi-chip setups require more automation and robustness in allparts of con�guration and runtime control. Hence, morepeople collaborate on di�erent aspects of the system which,in turn, introduces friction in the development and commis-sioning process. When the development of BrainScaleS OSstarted, we employed version control, personal interactionand test frameworks. Within the �rst four years of develop-ment, a chat system [22] —also utilized for users support—,continuous integration [23] and formalized code review [24]were added to the development process. At the time ofwriting, over 10 000 changes were submitted for discussion.We do not adopt a strict development process framework,e.g., like scrum, however, we do include ourselves withinthe agile movement. Weekly meetings provide the scopefor structured, long-term development, whereas our chatand code review systems encourage technical discussions of

details. The scenario is similar to CERN-style developmentwhere the developers are also to a large extent the users [25].Over 120 individuals contributed across various projects. Inthe following paragraphs, we introduce the key concepts.1) Open Source: Open Source software is a vital part in

almost all �elds of research. If not stated otherwise, thedeveloped libraries and tools are published at https://github.com/electronicvisions under the LGPL v2.1 [26] license. Wealso actively report bugs and push features upstream tothird-party libraries.

2) Software Design: The long-term software and hardwaredevelopment roadmaps are aligned to each other. Weeklysoftware development meetings form the basis of thecollaborative development. Problems and feature requestsare discussed as well as medium-term development planningis performed. If needed, smaller teams are formed to comeup with proposals that are then discussed in the plenum.However, the process is not fully democratic and at the endthe maintainers take the �nal decisions.

3) Review: The BrainScaleS hardware and software devel-opment teams adopted an explicit review-based developmentscheme. Tracking of the development history and thecurrent state of all components is handled by a set ofversion-controlled repositories. Developers propose changesto aspects of this state which are subsequently reviewed byother developers. At the end, the automated veri�cation ofeach change and an iterative review process result in a �nalversion which is then applied to the repository and becomesthe new current state. This enables a rolling release scheme.4) Veri�cation: Based on the ideas of the continuous

integration development methodology, the BSS veri�cationmethodology consists not only of software tests but also ofhardware-based as well as simulation-based tests. For eachproposed change the test result is fed back into the reviewsystem. When changes to software components are appliedto the current state, the modi�ed software is automaticallydeployed. Nightly tests serve as a measure for hardwareplatform health. The same experiment protocol can be usedon hardware and in a combined FPGA and digital chipsimulation. The latter is used for pre-tapeout veri�cationduring chip development.5) Software Environment: Reuse of existing software

packages reduces development costs but also introducestechnical debt in the form of dependencies on externalsoftware packages [27]. BrainScaleS uses a containerizedand explicit software dependency tracking system basedon singularity [28] and spack [29]. Updating the softwareenvironment is based on the same review and veri�cationsystem as before: developers propose a change to thedependency list, a testing container image is built and alltests are executed using this container image. If the containerhas build and code review as well as all tests have passed, theproposed change can be applied and the modi�ed containerimage becomes the new default container image.B. Foundations1) PyNN: PyNN [19] is a simulator-agnostic domain-

speci�c language for describing spiking neural network

models. Rooted in computational neuroscience, it focuses onthe initial network topology, model parameters and plasticityrules, de�nition of input stimulus and “recording” settings.

Matching our goal of an backend-independent experi-ment description language for spiking neural networks,we adopted PyNN as our high-level API. However, BSS-1 is not as �exible as a software simulator. For example,it only supports a �xed neuron model, limited-resolutionsynapses and a sparse connectivity matrix. Transforming auser-de�ned PyNN experiment into a similar, well-�ttinghardware con�guration is challenging task. In particular, itis a matter of neuron and synapse placement, spike routingand model parameter translation. Due of imperfections ofthe analog substrate and limited resources, like bandwidth,there will always be di�erences to the user-de�ned target.For a detailed study, see Brüderle et al. [20].

2) Programming Environment: GNU/Linux is a �exible andwell-supported host environment when developing customhardware. In addition, computational neuroscience reliesheavily on libraries and tools that are available for *NIX -likeoperating systems. Therefore, we only target Linux.

All core libraries are written in C++ with the exceptionof parts of the transport layer that need a tight coupling tothe Linux kernel and are therefore written in C.

We chose C++ because of several reasons that are notunique to an neuromorphic operating system but apply torequirements of large software suites that have at least amodest need for performance and robustness. It is a multi-paradigm strongly-typed compiled language which leads tothe discovery of many problems at compile time instead ofruntime. C++ supports many low-level manipulations thatare essential when directly communicating with a customhardware system. For example, in lower software layers thein-memory layout of data structures is required to matchthe formatting expected by the system. We always use thelatest language standard that typical open-source compilers,e.g., GCC [30] and LLVM [31] support.3) Python Wrapping: In experimental usage settings,

scripting languages o�er large advantages compared tocompiled languages. For example, the read–eval–print loop(REPL) allows for iterative testing of the hardware andalso for exploration of the software itself. Integration withthe broad Python ecosystem of scienti�c libraries, e.g.,numpy [32] or matplotlib [33], is an advantage in scienti�ce�ciency. Therefore, we support Python in addition toC++. To link the Python and the C++ world, we adopted afully automated wrapper code generation scheme based onpy++ and pygccxml [34]. The generated wrapper code usesboost::python [35]. Customizations of the wrapping processhave been collected in a library.4) Serialization: Serialization describes the process of

transforming in-memory data structures or object states intoa store and loadable format. This format can be written to,e.g., disk and loaded to restore the in-memory data structuresat a later point in time. Together with a framework forremote procedure calls, such as RCF [36], this allows forinter-process communication of higher-level data structures.

Though C++ does not o�er built-in support for serialization,it can be made available through several third party libraries.For BrainScaleS OS, we decided to use boost::serialization.

Listing 1: Example for boost serialization.class Spike{// ...private:template<typename Archiver>void serialize(Archiver& ar,

unsigned int const /*version*/){using boost::serialization::make_nvp;ar & make_nvp("addr", m_addr)

& make_nvp("time", m_time);}

addr_type m_addr;time_type m_time;

};

Listing 1 exempli�es the serialization of the two membervariables of the Spike class. More complex serializationfunctions are needed, e.g., when references and pointersare involved or di�erent versions should be considered tosupport long-term compatibility for pre-existing data sets.However, boost::serialization has excellent support for allthese scenarios. The on-disk format ranges from binary totext-based, such as JSON (custom extension) and XML.5) Utility Libraries:

a) Ranged enumeration types: In C++ numeric types donot have built-in support for range checks. Yet, it is bene�cialto have such concepts, since over and under runs can be athreat to both correctness and security. The rant [37] libraryprovides ranged integers and provides compile-time as wellas runtime checking of ranges. In case of compile-timestatements, violations produce compile errors; runtime errorsraise exceptions. Checks are implemented to be lightweightenough to be included in production code. If required, rangedtypes can be replaced by their native counterparts via acompile �ag to get rid of any remaining overhead. Rangedintegers are heavily used for the coordinates described in thenext section. Listing 2 demonstrates the use on the exampleof a ranged integer type.

Listing 2: Example for a ranged type.rant::integral_range<int, 0, 5> ranged_integer;

// e.g. 6 => throwsranged_integer = get_large_int();

// fails to compile (constexpr)ranged_integer = -1;

// worksranged_integer = 3;

b) Python-style C++ convenience library: pythonic++ [38]brings some Python-style programming to C++ for ease andmore expressive code. Listing 3 exempli�es this idea on theenumeration during the iterations over a vector.

Listing 3: The pythonic::enumerate function can be used to countthe iterations over an STL conform container.using namespace pythonic;

typedef std::vector<int> vec;

for (auto v : enumerate(vec{0, -1337, 42})){std::cout << v.first << " " << v.second << '\n';

}

BSS-1

FPGA

s and

CommunicationLayer

Map

and Ro

uteHardware Abstraction

Bit Formatting

PyNN Implementation

Calibration

Blacklisting

Samp

ling

Time-to-first SpikeRate-based Deep Learning

Fig. 2: The software stack as layers of abstraction. The core isthe neuromorphic hardware BSS-1. It is followed by physical andsoftware communication layers, i.e. the FPGAs and communicationlayer, followed by hardware abstraction and core functionality likemapping and routing. Next comes expert-level software used forblacklisting and calibration and the layer providing the PyNNabstraction. The last layer are the user-level applications andexperiments.

c) Bit Manipulation Library: bitter [39] provides acommon interface for bit operations on integral types aswell as std::bitset. Operations like reversal or cropping toranges are implemented.

III. ImplementationThe BrainScaleS OS consists of several software compo-

nent categories which are described in the following sections,see �g. 2 for an overview. At the end, the user is enabled todescribe and execute a neuromorphic experiment withoutdetailed knowledge of the underlying parts.

A. Con�guration

The correct con�guration of any hardware is a non-trivialtask, e.g., one needs to ensure to write the correctly formattedcommand to the correct memory location, while takinginto account other subtleties as, e.g., con�guration order. Inaddition, there can be a mismatch between the addressingof individual circuit instances and their physical or logicalplacement. Another aspect is con�guration timing as some

entities require settling times that have to be taken intoaccount. To address the issues raised, the BrainScaleS OScomprises of several libraries that allow for an easy andcorrect con�guration and control �ow.1) Coordinate System: In natural sciences, the proper

choice of a coordinate system often strongly contributesto a simple and clean solution for a problem. We argue thatthis applies also to the usage of hardware in general andin particular to neuromorphic computing. The myriad ofcomponents on a wafer lead to a con�guration space in doeorder of 50 MiB [20]. The memory for 4-bit weight and 4-bitaddress �lter of the 384 chips × 113k synapses per chip aloneamounts to 41 MiB per wafer. One needs a representationof those components in software [40].

Many symmetries in chip layout combined with wafer-scale integration naturally lead to abstraction on di�erentscales. Figure 3 gives an overview of a BSS-1 wafer andthe structure of its components. Framed in blue one cansee the layout of a single HICANN chip and its highdegree of self-similarity. In the background one silicon wafercontaining 384 of those chips is shown. This translationalsymmetry is re�ected in a hierarchical structure of thecoordinates. For each component we de�ne a coordinatewith the smallest granularity which then can be combinedto de�ne a higher hierarchical layer. We will illustrate thisexemplarily with the coordinate for neuron circuits. First, werepresent one neuron circuit on a chip: NeuronOnHICANN.This can then be combined with HICANNOnWafer resultingin NeuronOnWafer representing a speci�c neuron circuiton a wafer. Finally, NeuronGlobal can be composed fromNeuronOnWafer and Wafer to uniquely identify one neuroncircuit in the whole BSS-1 system. It is also possible tocast down to lower levels of representation, e.g., NeuronOn-Wafer::toNeuronOnHICANN(). Besides “lateral” conversionsbetween hierarchical layers it is also possible to translate“horizontally” among coordinates on the same level. Forexample SynapseOnHICANN::toNeuronOnHICANN() yieldsthe matching Neuron of a Synapse. See listing 4 for additionalexamples.

Another important feature is the possibility to create twodimensional grids that also have a notion of orientation,e.g., north and south. SynapseOnHICANN for exampleis structured in a grid of neurons per chip hemisphereand synapses per neuron. Grid coordinates also provideenumeration which is done in row-major order as shownin orange in �g. 3. Enumeration enables iteration of allcoordinates which is supported in both, C++ and Python,cf.listing 5 and listing 5. A string serialization exists thatserves as both, convenient short format for logging andfor argument parsing, cf. section III-C1. An example forthis functionality can be found in listing 7. The consistenceof this hierarchical structure is essential for a descriptive,reliable and maintainable low-level code base. About 80distinct coordinate types are used to describe elements of awafer module.

2) Bit Formatting: Typically, to con�gure a hardwareentity a pair of an address and its content is needed.

Listing 4: Example coordinate conversion.nrn = NeuronOnWafer(NeuronOnHICANN(Enum(5)),

HICANNOnWafer(Enum(5)))↪→

nrn.toNeuronOnHICANN()nrn.toHICANNOnWafer()

Listing 5: Example coordinate iteration in C++.for(auto nrn : iter_all<NeuronOnHICANN>()) {

std::cout << nrn << '\n';}

Listing 6: Example coordinate iteration in Python.for nrn in iter_all(NeuronOnHICANN):

print(nrn)

Listing 7: Example coordinate short formatting.h = HICANNGlobal(HICANNOnWafer(Enum(5)), Wafer(6))print(short_format(h))# W006H005print(from_string("W3"))# Wafer(3)

Fig. 3: Chip structure and coordinate system of a BSS-1 wafer: thebackground shows a silicon wafer with highlighted structures ofchips grouped in units of 4-by-2. The zoom-in shows one singleHICANN chip layout. Framed in white are various componentcategories. Black lines illustrate the structure of intra- and inter-chipevent buses. The row-major ordering scheme of a two-dimensionalcoordinate is shown in orange over a synapse array.

Addresses are used to identity writable and/or readablememory locations. The formatting of the content maydepend on various aspects, e.g., the entity’s physical locationon the chip. Therefore, functionality to address hardwareentities and to format bits is an essential part of thesoftware [41]. We use coordinates, section III-A1, to logicallyrepresent addresses. The content is represented as datastructures encapsulating functionality of the underlyingentity, cf. listing 8.

During development, for commission, expert use anddebugging, an iterative & interactive usage is facilitated byPython bindings for the lower-level con�guration functions.

Listing 8: Excerpt of the data structure SynapseDriver.struct SynapseDriver {public:

bool is_enabled() const { return enable; }...

private:bool enable = false...

};

Listing 9: Example for a pair of getter and setter functions of thestateless hardware abstraction layer.void set_synapse_driver(

Handle::HICANN& handle,SynapseDriverOnHICANN const& sdoh,SynapseDriver const& driver);

SynapseDriver get_synapse_driver(Handle::HICANN& handle,SynapseDriverOnHICANN const& sdoh);

This allows to, e.g., change parts of the con�guration —alsoout of order w.r.t. the canonical �ow— and directly observethe e�ects.

Listing 9 gives an example for a pair of getter andsetter functions. The handle represents the backend, eitheraccessing the hardware, the simulation backend describedin section III-B3, or other debug facilities. The coordinateSynapseDriverOnHICANN speci�es for which synapse driverthe settings in the SynapseDriver container driver should beapplied. The getter function is the same in reverse. Here,only the handle and the coordinate are passed. The bits readback from the hardware are decoded into and returned as aSynapseDriver object.3) High-level Con�guration: The core principal of the

con�guration of the neuromorphic hardware is that the user�rst speci�es the desired state to which then the hardwareis con�gured to [42]. Then, the hardware is con�gured tothis state. To facilitate this user-driven con�guration, allcon�gurable settings have functional names, e.g., the neuroncon�guration. By this, a viable level of “self-documentation”is achieved. The user-facing con�guration does not necessar-ily re�ect the exact granularity in which the hardware canbe con�gured, however, it does re�ect, as stated above, anachievable �nal state which then also allows validity checks.

Listing 10: Example for the stateful hardware abstraction layer.sthal::Wafer wafer;auto& hicann = wafer[HICANNOnWafer(Enum(5))];hicann.synapses[SynapseOnHICANN(Enum(123))].weight

= SynapseWeight(3);↪→

Listing 11: Example for per-FPGA parallelism via OpenMP.#pragma omp parallel for schedule(dynamic)for (size_t fpga_enum = 0; fpga_enum <

FPGAOnWafer::end; ++fpga_enum) {↪→

...}

Listing 10 demonstrates how to set the weight of a singlesynapse. The needed objects and bookkeeping structures arecreated on the �y. Also, checks on the availability database,

see section III-C1, are performed and raise exceptions if therequested resources are not available.

The con�guration is carried out with the maximumparallelism supported by the system, e.g., on a per-FPGA-basis with the help of OpenMP, see listing 11.

B. Control

1) Experiment Control Flow: The BSS-1 platform supportstwo distinct operation modes, both relying on FPGAs fordata I/O and for control �ow. Figure 4 illustrates the control�ow for the primarily used mode, the batch mode, whichsuits independent pre-de�ned experiments.

In either case, the �rst step is con�guring the neuromor-phic hardware. Many con�guration register accesses on thechip use a non-blocking access scheme requiring correcttiming. This is implemented by inserting wait instructionsbetween con�guration commands; the time intervals usea static worst-case timing model. It is also important tocon�gure hardware entities in a valid order. This is especiallyrelevant when entities are con�gured in parallel, e.g., perFPGA. Synchronization barriers must then be added to thecon�guration �ow so that only when all entities have reacheda certain stage, con�guration is continued. Another point totake care of is enabling triggers for, e.g., the recording ofanalog membrane traces which are supposed to start withthe experiment; it is the point in time when the stimulusbegins and recording of spike events is enabled.

Timed spike event release as well es recording is handledby 48 FPGAs on each wafer module. Each FPGA has accessesto 1.25 GiB of DRAM providing bu�er memory for, e.g., inputstimulus and recorded data. In case of the batch mode, thecomplete input to the network is prede�ned on the host,then sent to the FPGA and released upon experiment start.Simultaneously, spikes generated by the neuromorphic chipsare recorded. At the end of the experiment, the recordedspikes and analog membrane traces are sent back to host. Onetypical application of this mode are deep neural networkswhere synaptic weights are optimized by an o�ine learningalgorithm. If the hardware is allocated for a longer timeperiod, the experiment framework also supports selectableautomatic di�erential con�guration reducing con�gurationoverhead in later iterations.

In the other so called hybrid operation mode parts ofthe network or a virtual environment are simulated on thecontrol cluster and interact with the spiking neural networkrunning on BSS-1. This mode of operation is also known asreal-time closed-loop. Control �ow di�ers compared to theaforementioned mode as spike events sent by the host arenot pre-bu�ered and timed by the FPGA but instead, theyare directly injected into the chip upon arrival. Vice versa,emitted spikes from the network are directly sent to hostand reacted upon. The challenge is to match the accelerationfactor in both realms and keep the latency of the networkcommunication as low as possible, see section III-B2.

2) Communication Layer: There are two main categories ofdata which need to be transferred between the neuromorphichardware and conventional compute nodes. On the one hand

FPGA

staticconfiguration

Host

spike-trainupload

end ofconfiguration

spike-traindownload

end ofexperiment

start ofexperiment

experimentruntime

inputstimulus(spikes)

Fig. 4: Control �ow of a typical experiment in batch mode. Blackboxes indicate activity of host and FPGA during the di�erent steps.

there is con�guration data, e.g., neuron parameters, networktopology, and on the other hand the activity of the network,i.e. spike events. Due to the accelerated operation of theBSS-1 system typical neuron activities of O(100 Hz) result inon-wafer event rates exceeding Tera-Events per second. Thisdemands high-throughput data exchange between BSS-1 andcluster control nodes. Ethernet was chosen as conventionaldata network equipment is readily available and, at the timeof writing, commercial hardware supports bandwidths of upto 100 GiB/s. On BSS-1, the external connectivity is providedby 48 1-Gigabit Ethernet links. However, this bandwidth isstill not su�cient to completely monitor the aforementionedon-wafer activity. The FPGAs provide an additional bu�erstage for input and output data, but �ltering and selectingin- and outputs is still needed. In the case of a deep neuralnetwork this would for example simply be the in- and outputlayers.

Furthermore, transfer of data, especially con�gurationdata, needs to be robust. For Ethernet-based communicationthe Transmission Control Protocol (TCP) on top of theInternet Protocol (IP) is most commonly used as a reliablesecure transport layer protocol. At the time of developmentthere where no open source FPGA implementations of TCPavailable and even now available solutions are very resourcedemanding [43]. Hence, the BSS-1 FPGAs implement acustom sliding-window protocol with an automatic resendmechanism (ARQ) on top of the unreliable User DatagramProtocol (UDP) over IP. The software implementation hasbeen open-sourced in the past [44]. Additional features anal-ogous to congestion control, like roundtrip-time estimationas well as the slow start algorithm have been implementedin both, software and hardware.

However, the hybrid operation mode, cf. section III-B1,demands low-latency and low-jitter transport of spike events;con�guration data is still transmitted via the reliable customtransport layer protocol but spike events are transferredbest-e�ort facilitated by memory-mapped zero-copy receiveand transmit ring bu�ers based on PACKET_MMAP [45].

Additional measures like setting CPU core a�nity aretaken to reduce jitter to a minimum on the host side.

3) Hardware Simulation: The so-called “executable systemspeci�cation” [46] (ESS) is a hardware simulator of theBSS-1 system implemented in C++/SystemC. It containsbehavioral, timing-accurate models of the digital components

and functional models of the analog neural components,e.g., the hardware neurons are numerically simulated AdExneurons. O�ering the same con�guration interface as thereal hardware and being fully executable, the ESS has beenessential for the hardware-software co-design [20] and stillserves as a validation tool for the software stack, especiallyfor the mapping, con�guration and experiment executionsteps. In addition, the ESS allows to evaluate the e�ect ofBSS-1 design-speci�c constraints (e.g., limited stimulationand recording bandwidth, spike time jitter, reduced parameterresolution) on neuromorphic experiments in isolation fromdistortions due to the mismatch of the mixed-signal circuits.For a detailed study see Petrovici et al. [47].

C. Conditions Support

Wafer-scale hardware operates under the assumption thatindividual components can be switched o� and circumvented.In addition, the analog nature makes it necessary to, atleast, apply a working point calibration. For this, conditionssupport libraries are put in place and described below.

1) Availability Database: Errors during the manufacturingprocess and the assembly of the wafer lead to varyingconditions of individual components. Moreover, modifyinghardware parameters may lead to a change in the responseof these components. Disregarded, they might either distortsimulation results or make the execution of experimentsimpossible in the �rst place. Consequently, it is mandatoryto be aware of the state of the components and handleit dynamically. Therefore, the availability database wasdeveloped [48].

Combined with digital tests, cf. section IV-C1, this allowsfor storing and handling of the used components. It isimplemented in C++ and uses XML with boost::serializationas the storage backend. Based on the coordinate system itstores a sparse representation of the �agged componentswithout the notion of reasons as a whitelist or a blacklist.By this, the natural hierarchy of the system is mapped tothe database. Thus, e.g., HICANNOnWafer �ags the full chipand NeuronOnHICANN �ags only a single neuron circuit ofthe chip. Subsequently, using the database, other parts ofthe software can simply avoid the �agged components.

This also allows for the second use case of the availabilitydatabase. Components can be marked arti�cially as notavailable to manipulate the hardware resources of an exper-iment without the requirement of an additional interface.Python bindings allow to construct convenience tools likea command line interface, cf. listing 12. As a result, a perexperiment set of usable components can be generated andadapted dynamically.

Listing 12: Availability database command line interface.redman_cli.py . W33H0 has neuron 0# True

redman_cli.py . W33H0 disable neuron 1

2) Parameter Translation and Calibration Database: Mi-croelectronics’ manufacture deals with non-uniformities inthe circuits produced across a silicon wafer. These transistormismatches result in varying response from neuron toneuron circuit. To compensate, a calibration framework [49]maps high-level parameters to the hardware parameter space,homogenizing the response of neuron circuits.

First, the biological units are converted to the hardwarecompatible range. For time constants, the acceleration factorα = 1000 . . . 10 000 is taken into account:

τhardware = α · τbiology.

Voltages also need to be scaled by s and shifted by o,respectively:

Vhardware = s · Vbiology + o,

with typical values s = 10 and o = 1.2V Similar conversionsare needed for synaptic weights.

Now that the desired hardware values are known inphysical units, the conversion to the digital domain canhappen. This step does two things. The translation fromphysical units to digital units while at the same time takinginto account variations from circuit to circuit, i.e. it appliescalibration data. For this, the calibration database allowsto store parameters for a set of pre-de�ned functions, e.g.,polynomials. In addition, the transformation classes providea numerical function inversion.

Also, the input values can be checked to lie within agiven range of validity. The returned value can then beeither clipped, an exception can be thrown or the validityrange can be ignored.

Listing 13 demonstrates the usage of the library on theexample of a linear function.

Listing 13: Example for a linear calibration function.// linear transformation from, e.g., 0 - 1.8 V to 0

- 1023 DAC↪→

Polynomial linear({0.0, 1023./1.8}, 0.0, 1.8);linear.apply(0.9);// 511.5linear.reverseApply(256);// 0.450linear.apply(2); // defaults to clipping// 1023

D. Network Description

1) PyNN Interface: We implement the PyNN -API as a thinC++ library for which Python bindings are generated [50].Compared to a Python-based implementation, this allowsfor a memory-e�cient handling of larger data sets suchas weight matrices of large neural networks, stimulus orrecorded data. For the user, however, it appears like anyother PyNN implementation, e.g., PyNN.nest. Internally, ittranslates from the PyNN ’s imperative experiment descrip-tion to an object-oriented description in the underlyingC++ layer. The individual elements, e.g., populations andprojections, are similar in their structure to PyNN. However,if found necessary, we restructure the data to our liking as

it is decoupled from the user-facing API. Also, pure C++usage is supported in a structured way.

2) Map & Route: Mapping and routing of neural networksdescribed in PyNN to the neuromorphic hardware is a non-trivial task. The complexity and scope of the problem issimilar to the synthesis of FPGA bit�les. Therefore, theprocess is only described brie�y. The full implementationcan be found at [51]. Also, the map & route implemen-tation undergoes substantial changes as new features andimprovements are being developed.

In its simplest form, we implement a greedy strategywithout back tracking. First, we place neurons from PyNNpopulations to hardware neuron circuits. Hereby, di�erentneurons may be represented by a di�erent number of neuroncircuits on the hardware. Insertion points for spike inputfrom the FPGAs are placed as well. The user has the optionto constrain the automatic placement of neurons and spikesources. User parametrization is facilitated by a custom classthat works aside from PyNN. Listing 14 shows how a user canrestrict the placement of a population to a certain HICANNor to a list of allowed options.

Listing 14: Example for constraining placement.pop = pynn.Population(...)marocco.manual_placement.on_hicann(pop,

HICANNOnWafer(Enum(42)))↪→

pop2 = pynn.Population(...)marocco.manual_placement.on_hicann(pop2,

[HICANNOnWafer(Enum(23)),HICANNOnWafer(Enum(24))])

↪→

↪→

Listing 15: Example for querying a mapping result, where thehardware neurons corresponding to PyNN neurons are retrieved.pop = pynn.Population(5, ...)for pynn_neuron in enumerate(pop):items =

runtime.results().placement.find(pynn_neuron)↪→

for item in items:for hardware_neuron in item.logical_neuron():

...

After the placement of neurons, the PyNN projectionsare transformed into synapses and on-chip routes on thehardware. This is the most time-consuming step as severalhardware constraints must be taken into account, e.g., thelimited number of allowed switches per route.

Listing 15 shows how to retrieve information on theallocated hardware after the placement: the hardware neuroncircuits are looked up for all neurons of a PyNN population.This is useful for, e.g., directly manipulating low-levelhardware parameters. The link between PyNN and the resultof the map & route step is stored into an intermediaterepresentation format, cf. �g. 5.

The on-chip bus network is represented as a graph usingthe boost::graph [52] library, where bus lines are verticesand switches are edges. During the creation of the graph,hardware availability data is already taken into account,i.e. hardware components that should not be used are notincluded in the graph representation. On-wafer routes can

Fig. 5: The transformation of PyNN to a hardware con�guration(container) makes use of intermediate representations (IR). The IRalso links PyNN and hardware entities and allows for look-ups inboth directions.

Fig. 6: Screenshot of the web-based visualization. Chips are coloredwith increasing opacity proportional to the number of placedneurons. On-chip routes are also colored and can be click-selectedto reveal more details.

be found by custom traversal algorithms of the graph or byusing graph search algorithms like Dijkstra.

Being able to have a visual representation of the foundhardware con�guration is important for both, debuggingand understanding possible improvements of manual orautomatic placement. For this, a web-based visualizationhas been developed. Based on previous e�orts to buildvisualization tools, the main requirement is to not replicateany code paths that are already part of the software stack.Another requirement is the possibility to run the toollocally and standalone, i.e. without the need for a serverand the availability of the full software stack. This ispossible by transpiling parts of the BrainScaleS OS C++libraries to JavaScript, including classes representing themap & route intermediate representation and its serializationimplementation. Now, only the transpiled JavaScript librariesand the output of the mapping must be at hand. The top-levelcode is written in TypeScript [53].

We rely on Pixi.JS [54] for a fast 2D graphics enginesupporting WebGL. It is capable of rendering large networkswith many details of the hardware con�guration. The toolo�ers di�erent levels of details where, e.g., by zooming inall used synapses and neurons become visible. An exampleis shown in �g. 6.

IV. OperationA. Resource Management and System Access

BrainScaleS neuromorphic platform resources are time-shared and partitioned between multiple experiments and/orusers. In contrast to typical digital systems, analog neuro-morphic hardware substrates are not homogeneous. Usersneed to be able to request speci�c hardware instanceswhen running experiments. We use SLURM [55] —a HPCjob scheduler— to handle resource requests for hardwarecomponents.SLURM was extended utilizing its plugin API to handle

various requirements related to inhomogeneous hardwareresources. Being a mixed-signal neuromorphic system, in-dividual BrainScaleS systems behave slightly di�erentlywhich is why experimenters need a way to explicitlyspecify individual resource instances. The coordinate systemdescribed in section III-A1 is used to provide a familiarinterface to the user. Di�erent hardware components havevarying degrees of granularity that can overlap and haveinterdependencies. We allocate the smallest needed subsetof resources inferred from the user request. In principle, theEthernet-based communication described in section III-B2allows access to each FPGA from any conventional computenode in the same network. To prevent accidental clashesbetween concurrent experiment runs we separate individualBSS-1 modules into IPv4 subnets and deny access based ondefault �rewall settings. When a user speci�es a hardwareresource for a SLURM job a �rewall rule to accept tra�c isautomatically added during job runtime. Experiment softwarealso compares its own resources with the allocated SLURMresources to detect possible mismatch.

On top of the direct access to the system as explainedabove, we provide access via HBP’s collab infrastructure [56].Jobs are fetched from the HBP neuromorphic platformqueuing service with the help of [57] and passed on toSLURM. Every few seconds, the job states of our schedulerand the upstream queue manager are synchronized.

B. Monitoring

Managing a large complex hardware system is unfeasiblewithout extensive monitoring, as malfunction of any individ-ual component can be fatal for operation. Monitoring cangenerally be split into three steps: aggregation, storage andvisualization. Likewise there are two di�erent types of datato be gathered, time-series data (e.g., voltage, temperature)and event data (e.g., powering o� components).

The general �ow for monitoring data of a wafer moduleis shown in �g. 7. There are around 1200 time-seriesdata sources within one wafer module. Important sensorslike wafer temperature are read out every few seconds.Additionally, events for powering parts on/o� or alerts aregenerated. The data aggregation is performed on a RaspberryPi via a software daemon handling several communicationschannels, e.g., I2C. On the Raspberry Pi a �rst data analysisis done in order to have a quick response to a dangeroussystem state. For example the temperatures are checked tobe in an allowed range, above a given threshold the system

Wafer Module Control RPi

Compute Nodes

Monitoring Server

Ganglia

Syslog

Elasticsearch

Graphite Grafana

Kibana

custom daemonsensors

Aggregation Storage Visualization

Fig. 7: Flow of monitoring data from aggregation to storage andvisualization. Grey boxes represent involved devices on which thecorresponding software libraries run. Software responsible for time-series data is shown in yellow and for event data in green. Arrowsillustrate connectivity between the di�erent components.

is turned o�. Furthermore, all microcontrollers providingdata to the Raspberry Pi perform local data evaluation testsand, therefore, detect false states faster than the RaspberryPi. Time-series data is stored on a central Carbon [58] serveroutside the Raspberry Pi. For the conventional computenodes we use Ganglia [59] for data aggregation which alsofeeds into the Graphite data base. Graphite uses a round-robin database for automatic data compression after certainintervals. Event aggregation is done utilizing syslog [60]which is parsed by Logstash [61]. Filtered events are storedin an Elasticsearch [62] database. Grafana [63] is usedto visualized time-series data. It allows the creation ofdashboards which give insight to the state of the systemon various levels of detail. This facilitates getting a quickoverview of relevant data from wafer modules and the stateof the conventional compute cluster while simultaneouslyallowing to drill down for more details. Additionally, eventslike powering on components can also be shown in Grafanato easily link events and changes in time-series data. Ingeneral we use Kibana [64] to visualize event data.

C. Commissioning

1) Digital memory tests: In large complex hardwaresystems, variations of individual components are inevitable.As already mentioned in section III-C1 the behavior of thesecomponents change due to varying hardware parameterssuch as supply voltage and might disturb the execution ofexperiments. As a result, it is important to keep track of thestate of the components and be aware of it during experimentexecution. This is achieved by digital memory tests that areexecuted after assembly as well as periodically. Here, for aspeci�c hardware con�guration given by, e.g., supply voltageand clock frequency, each digital memory of every HICANNis read/write-tested with random values. The results arecompared and if a malfunctioning component is found itis �agged in the availability management database. Thedatabase re�ects the hierarchical structure of the hardware,so that always the largest functional unit that exclusivelydepends on the malfunctioning components is �agged, shownin �g. 8. The information can then be extended individuallyfor each experiment and stored by serializing the updateddatabase to disk, which is typically an XML-based �le format,cf. section III-C1. During experiment execution this data is

Fig. 8: Digital test of malfunctioning components highlighted ingrey. After the test (right side) these components are markedas not available using the hierarchy of the system. As a result,individual components up to large functional units, consisting ofmany components, are marked as not available.

Fig. 9: Three neurons set for continuous spiking activity exhibitnon-uniform threshold voltage under the same �oating gatecon�guration. After applying the per-neuron calibration, parameterslike Vtreshold can be set accurately across di�erent neuron circuits(Traces were hand-drawn for illustrative purposes, with attentiononly to Vthreshold).

then deserialized, which allows for skipping the unavailablecomponents. Besides the experiment execution, the digitalmemory tests are also used in continuous integration tomonitor and store the state of the hardware.

2) Calibration: The one-time circuit characterization [65]runs sequences of experiments that sweep the neuronparameters (stored as 10-bit values in analog �oating gates onthe HICANN), measures the changes’ impacts, and employsdi�erent �ts depending on the parameter e�ect’s response.A calibration database is then �lled with the transformationdata for its utilization on routine hardware usage. Thee�ect of applying such parameter mapping to the neuroncon�guration is exempli�ed in �g. 9.

V. ApplicationsThe previous sections motivated and detailed the status

of the BrainScaleS OS. In the following, �rst a minimalexperiment is demonstrated with the key concepts inaction. It is followed by examples for more complex “full”experiments.

A. A Minimal Experiment

Listing 16: Example for an experiment.import numpy as np

# BrainScaleS OS importsimport pyhmf as pynnimport pyhalco_hicann_v2 as Cfrom pyhalco_common import Enumimport pyhalbeimport pysthalfrom pysthal.command_line_util import init_loggerfrom pymarocco import PyMaroccofrom pymarocco.runtime import Runtimefrom pymarocco.results import Marocco

init_logger("WARN", [])

marocco = PyMarocco()runtime = Runtime(C.Wafer(33))

pynn.setup(marocco=marocco, marocco_runtime=runtime)

experiment_duration = 1000 # ms

neuron_parameters = {'cm': 0.2, # nF'v_reset': -20., # mV'v_rest': -20., # mV'v_thresh': 100, # mV'e_rev_I': -20., # mV'e_rev_E': 0., # mV'tau_m': 10., # ms'tau_refrac': 0.1, # ms'tau_syn_E': 2., # ms'tau_syn_I': 5., # ms

}

pop = pynn.Population(2, pynn.IF_cond_exp,neuron_parameters)↪→

stimulus = pynn.Population(1, pynn.SpikeSourceArray, {'spike_times': [0, 5, 10]})

# choose a specific HICANN for given pyNN populationmarocco.manual_placement.on_hicann(pop,

C.HICANNOnWafer(Enum(0)))↪→

# record both, neuron membrane trace and spikespop.record()pynn.PopulationView(pop).record_v()

# connect the stimulusproj = pynn.Projection(stimulus, pop,

pynn.AllToAllConnector(weights=0.001),target='excitatory')

↪→↪→

# perform mapping but do not execute on hardwaremarocco.backend = PyMarocco.Nonepynn.run(experiment_duration)

# hardware-level settingssynapses_result =

runtime.results().synapse_routing.synapses()↪→proj_items = synapses_result.find(proj)for proj_item in proj_items:

syn_c = proj_item.hardware_synapse()hicann = runtime.wafer()[syn_c.toHICANNOnWafer()]synapse = hicann.synapses[syn_c.toSynapseOnHICANN()]synapse.weight = pyhalbe.HICANN.SynapseWeight(3)

# run on hardwaremarocco.skip_mapping = Truemarocco.backend = PyMarocco.Hardwarepynn.run(experiment_duration)

np.savetxt("membrane.txt", pop.get_v())np.savetxt("spikes.txt", pop.getSpikes())

Listing 16 shows an example experiment. It demonstratesall software features discussed in the previous sections. First,a couple of Python modules are imported. The maroccoobject is instantiated that allows for custom con�gurationsthat are not part of the PyNN API. The runtime objectholds, amongst others, the sthal representation of the wafercon�guration that will be used for low-level re-con�guration.Next, parameters like the experiment duration and neuronparameters are set as variables. A population of neurons aswell as a stimulus are created with two and one neuron,respectively. The population of neurons is placed on aspeci�c HICANN. If manual placement is not given, themapping software will �nd a location depending on thechosen mapping algorithm. No mapping hint is given forthe stimulus. It will be inserted as close as possible to themapped neuron population while adhering to bandwidthlimitations as good as possible. The population is then askedto record both, its spikes and membrane potential. Next, aprojection is drawn between the stimulus and the neurons.The projection is stored in a variable for later lookup.

Now that the network is completely setup, the mappingcan be carried out, but it is not yet executed (backend=None).By this, the user can look up the hardware synapse betweenthe stimulus and the neurons. Doing so we set manuallya digital weight of 3. Then we skip the mapping, setthe backend to hardware and execute. After pynn.run theresulting membrane trace and spikes can be read out.

Listing 17: Example for an experiment invocation.# allocate the full modulesrun -p experiment --wafer 33 experiment_example.py

# allocate only HICANN 0 (with analog readout andtrigger by default)↪→

srun -p experiment --wafer 33 --hicann 0experiment_example.py↪→

The network execution is then invoked by calling listing 16with a SLURM command like srun, see listing 17. The systemon which the experiment is conducted is given as well asthe partition which is used for accounting and priority.

B. Examples for Full Experiments

In the simple example explained above, the split betweenmapping and execution is only necessary if low-level accessis wished. However, an important application are chip-in-the-loop experiments where it is crucial that an iterativere-con�guration is possible to, e.g., compensate for trial-to-trial variations.

An experiment where this was used is detailed in [10] fortraining of a deep network for digit classi�cation. Figure 10shows the concept. After training an arti�cial neural network,the weights of a matching hardware network are setaccordingly. However, due to both, trial-to-trial variationsand di�ering responses of the arti�cial w.r.t. the hardwareneurons, the classi�cation performance is diminished. Bycontinuing the training in the loop, the performance canbe restored. For this, the response of all neurons in the

Fig. 10: Each iteration of in-the-loop training consists of two passes.In the forward pass, the output �ring rates of the LIF network aremeasured in hardware. In the backward pass, these rates are used toupdate the synaptic weights of the LIF network by computing thecorresponding weight updates in the ReLU network and mappingthem back to the hardware. Adapted from Schmitt et al. [10].

hardware network is fed back into the training loop of thearti�cial network.

Accelerated physical emulation of Bayesian inferencein spiking neural networks was demonstrated in Kunglet al. [66] where the full BrainScaleS OS was used aswell. A network of spiking neurons was set up to samplefrom a Boltzmann distribution. The network was alsotrained iteratively, however, not in companion with anarti�cial network. Classi�cation and pattern completion weredemonstrated on two datasets.

Another example is Göltz et al. [67] that demonstratesclassi�cation based on spike timing only. Again, the hardwarein-the-loop approach was used to train a network classifyingimages on BSS-1.

VI. Future DevelopmentsA. Separation of Experiment Con�guration and Execution

The experiment demonstrated in section V-A executed themapping and the neuromorphic emulation in one process.Most importantly, the requested hardware resources had tobe speci�ed prior to the mapping. This is an unfortunateorder as it does not allow for, e.g., choosing the systemdynamically or specifying only a subset of the wafer forrunning experiments in parallel. However, the necessaryingredients to overcome this problem are in place andwork is currently carried out to implement a solution basedon the serialization capabilities of the data structures, seesection II-B4.

B. Next-generation Python Binding Generation

The design and development of the BrainScaleS OS startedin 2009. In the meantime, several external dependencies havebeen deprecated. In particular, our auto-generated Pythonwrapping depends on gccxml where development stoppedin 2015. It depends on gcc ≤ 4.9.3 blocking the usage ofthe latest C++ features from the 14, 17 and 20 standards inheader �les. We evaluated several approaches, including theusage of LLVM’s castxml, but resorted to developing a new

wrapper code generator —genpybind [68]— which is basedon LLVM libraries. The transition from the py++-based togenpybind is now in progress. In addition, genpybind alsoattacks binding generation from a di�erent angle with amore �ne-grained and explicit approach.

C. Towards BrainScaleS-2

Software development for BSS-2 started in 2016 and buildsupon the results —the BrainScaleS OS— presented in thiswork. We try to re-use and adapt as much as possible fromthe existing code base. Especially the coordinate system, cf.section III-A1, has proven bene�cial. However, early in thedesign phase we decided to improve the hardware abstractionlayers by introducing structured types encapsulating all on-chip and on-FPGA con�gurable hardware entities. Thesetypes also provide explicit implementations for encodingto and decoding from hardware con�guration bitstreams.In conjunction with a timer-based execution �ow on theFPGA, this allows for experiments being described as antimed event sequence. The C++ API makes heavy useof std::future-like interfaces to expose an asynchronousinterface to the experiment control �ow. Additionally, afast experiment scheduler has been developed for BSS-2.It allows for approximately ten experiments per secondwhere —due to the neuromorphic speed-up factor— eachexperiment represents up to 100 s of emulated time in thebiological model. Based on this, restructuring work on BSS-1started and a corresponding implementation was developed.Similarly, the genpybind tool was created during BSS-2 low-level software development. See Müller et al. [69] for adetailed description.

VII. Discussion

This work describes the latest version of BrainScaleS OS,the software stack operating the BSS-1 platform. It allowsto accomplish the main goal of the wafer-scale mixed-signalneuromorphic system BrainScaleS-1: designing and runningwafer-scale experiments. The software stack aims for non-expert usage, e.g., by neuroscientists, while maintainingaccess to all other abstraction levels for expert users. Wegive a detailed overview of the individual software com-ponents and describe di�erent aspects. From the hardwarecon�guration, over the interaction with the system, e.g.,setup, runtime control and result read out. We describethe transformation of user-de�ned experiments into a validhardware con�guration, as well as the necessary resourcemanagement and monitoring.

BrainScaleS adopted development methodologies and toolsoriginating in software engineering to improve platformrobustness and experiment reproducibility. BSS-1 is operatedas a platform which is available for the research community.

Several experiments [12, 66, 67] demonstrate thatBrainScaleS OS is a viable basis for using BSS-1. In addition tothe publications, several thesis in our group made use of it forconducting neuromorphic experiments and commissioningwork.

VIII. ContributionsE. Müller is the lead developer and architect of the

BrainScaleS software stack. S. Schmitt contributed to thecalibration, the stateful con�guration layer and the generalusage �ow. C. Mauch contributed to the system con�gurationlayers as well as system operation. S. Billaudelle is amain contributor to the BSS-1 PyNN API implementation.A. Grübl contributed software for low-level con�gurationas well as the system simulation backend. M. Güttler isthe main developer of the system-level operation software,e.g., system monitoring and controlling, and contributedto low-level �rmware. D. Husmann is the main developerof the system-level test software suite and contributed tosystem-level operation software. J. Ilmberger contributed tothe communication layer and the analog readout framework.S. Jeltsch is the main developer of the map & route layer.J. Kaiser contributed to speed-up the synapse con�guration.J. Klähn contributed to the map & route layer. M. Kleidercontributed to the calibration of the system. C. Koke is themain developer of the stateful con�guration layer and thecalibration framework. J. Montes contributed to calibrationscalability. P. Müller evaluated the performance of theBSS-1 neuromorphic circuit implementation. J. Partzschcontributed software for the low-level system con�guration.F. Passenberg optimized the map & route algorithms toenable successful topology mapping to wafer moduleswith non-ideal blacklisting state. H. Schmidt contributedto digital blacklisting. B. Vogginger is a main developerto the simulation backend and contributed to the map& route layer. J. Weidner contributed to the web-basedcon�guration visualization and acquired con�guration resultsfor the Jülich cortical column network. C. Mayr contributedto the system design (hardware and software) of the o�-wafer communication stack. J. Schemmel is the lead designerand architect of the BrainScaleS-1 neuromorphic system. Allauthors discussed and contributed to the manuscript.

AcknowledgmentsThe authors wish to thank all present and former members

of the Electronic Vision(s) research group contributing tothe BSS-1 hardware system, development and operationmethodologies, as well as software development. The au-thors express their special gratitude towards: 1) DanielBarley for his contribution to the parallel ADC-readoutsoftware; 2) Richard Boell for his contribution to the web-based visualization tool; 3) Patrick Häussermann for hiscontribution to experiment isolation in the scheduler; 4) KaiHusmann for his contribution to the low-level system controlenvironment; 5) Lukas Pilz for his contribution to evaluatesupport for iterative con�guration; 6) Vitali Karasenko for hiscontribution to the graphics; 7) Alexander Kononov for hise�ort when commissioning the HICANN chip; 8) DanielKutny for his contribution to the monitoring solution;9) Dominik Schmidt for his contribution to the calibrationframework; 10) Moritz Schilling for his contribution to theinitial implementation of the secure transport layer protocol;11) Andreas Baumbach, 12) Oliver Breitwieser and 13) Yannik

Stradmann for their work on continuous integration andstructured deployment of the software environment. Weespecially express our gratefulness to the late KarlheinzMeier who initiated and led the project for most if its time.

This work has received funding from the EU ([FP7/2007-2013], [H2020/2014-2020]) under grant agreements 604102(HBP), 269921 (BrainScaleS), 243914 (Brain-i-Nets), 720270(HBP) and 785907 (HBP) as well as from the Manfred StärkFoundation.

References[1] Steve B. Furber, David R. Lester, Luis A. Plana, et al.

“Overview of the SpiNNaker System Architecture”. In:IEEE Transactions on Computers 99.PrePrints (2012).issn: 0018-9340. doi: http://doi.ieeecomputersociety.org/10.1109/TC.2012.142.

[2] Thomas Pfeil, Andreas Grübl, Sebastian Jeltsch, et al.“Six networks on a universal neuromorphic computingsubstrate”. In: Frontiers in Neuroscience 7 (2013), p. 11.issn: 1662-453X. doi: 10.3389/fnins.2013.00011. url:http : / / www. frontiersin . org / neuromorphic % 5C _engineering/10.3389/fnins.2013.00011/abstract.

[3] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, et al.“Loihi: A neuromorphic manycore processor with on-chip learning”. In: IEEE Micro 38.1 (2018), pp. 82–99.

[4] Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza,et al. “A million spiking-neuron integrated circuit witha scalable communication network and interface”. In:Science 345.6197 (2014), pp. 668–673.

[5] Johannes Schemmel, Daniel Brüderle, Andreas Grübl,et al. “A Wafer-Scale Neuromorphic Hardware Systemfor Large-Scale Neural Modeling”. In: Proceedings ofthe 2010 IEEE International Symposium on Circuits andSystems (ISCAS). 2010, pp. 1947–1950.

[6] C. A. Mead. “Neuromorphic Electronic Systems”. In:Proceedings of the IEEE 78 (1990), pp. 1629–1636.

[7] S. Moradi and G. Indiveri. “An Event-Based NeuralNetwork Architecture With an Asynchronous Pro-grammable Synaptic Memory”. In: IEEE Transactionson Biomedical Circuits and Systems 8.1 (Feb. 2014),pp. 98–107. issn: 1940-9990. doi: 10.1109/TBCAS.2013.2255873.

[8] J. Schemmel, A. Grübl, K. Meier, et al. “ImplementingSynaptic Plasticity in a VLSI Spiking Neural NetworkModel”. In: Proceedings of the 2006 International JointConference on Neural Networks (IJCNN). IEEE Press,2006.

[9] Sebastian Millner, Andreas Grübl, Karlheinz Meier,et al. “A VLSI Implementation of the AdaptiveExponential Integrate-and-Fire Neuron Model”. In:Advances in Neural Information Processing Systems 23.Ed. by J. La�erty, C. K. I. Williams, J. Shawe-Taylor,et al. 2010, pp. 1642–1650.

[10] Sebastian Schmitt, Johann Klähn, Guillaume Bellec,et al. “Classi�cation With Deep Neural Networkson an Accelerated Analog Neuromorphic System”.In: Proceedings of the 2017 IEEE International Joint

Conference on Neural Networks (2017). doi: 10.1109/IJCNN.2017.7966125. url: http://ieeexplore.ieee.org/document/7966125/.

[11] Chetan Singh Thakur, Jamal Lottier Molin, GertCauwenberghs, et al. “Large-Scale Neuromorphic Spik-ing Array Processors: A Quest to Mimic the Brain”.In: Frontiers in Neuroscience 12 (2018), p. 891. issn:1662-453X. doi: 10.3389/fnins.2018.00891. url: https://www.frontiersin.org/article/10.3389/fnins.2018.00891.

[12] Sebastian Schmitt, Johann Klähn, Guillaume Bellec,et al. “Classi�cation With Deep Neural Networkson an Accelerated Analog Neuromorphic System”.In: Proceedings of the 2017 IEEE International JointConference on Neural Networks (2017). doi: 10.1109/IJCNN.2017.7966125. url: http://ieeexplore.ieee.org/document/7966125/.

[13] Oliver Rhodes, Petruţ A. Bogdan, Christian Bren-ninkmeijer, et al. “sPyNNaker: A Software Packagefor Running PyNN Simulations on SpiNNaker”. In:Frontiers in Neuroscience 12 (2018), p. 816. issn: 1662-453X. doi: 10.3389/fnins.2018.00816. url: https://www.frontiersin.org/article/10.3389/fnins.2018.00816.

[14] Andrew G. D. Rowley, Christian Brenninkmeijer,Simon Davidson, et al. “SpiNNTools: The ExecutionEngine for the SpiNNaker Platform”. In: Frontiers inNeuroscience 13 (2019), p. 231. issn: 1662-453X. doi:10.3389/fnins.2019.00231. url: https://www.frontiersin.org/article/10.3389/fnins.2019.00231.

[15] Chit-Kwan Lin, Andreas Wild, Gautham N Chinya, etal. “Programming Spiking Neural Networks on Intel’sLoihi”. In: Computer 51.3 (2018), pp. 52–61.

[16] Arnon Amir, Pallab Datta, William P Risk, et al. “Cog-nitive Computing Programming Paradigm: A CoreletLanguage for Composing Networks of NeurosynapticCores”. In: The 2013 International Joint Conference onNeural Networks (IJCNN). IEEE. 2013, pp. 1–10.

[17] Catherine D. Schuman, Thomas E. Potok, Robert M.Patton, et al. A Survey of Neuromorphic Computingand Neural Networks in Hardware. 2017. eprint: arXiv:1705.06963.

[18] Daniel Brüderle, Eric Müller, Andrew Davison, et al.“Establishing a novel modeling tool: a python-basedinterface for a neuromorphic hardware system”. In:Frontiers in Neuroinformatics 3 (2009), p. 17. issn: 1662-5196. doi: 10.3389/neuro.11.017.2009. url: https://www.frontiersin.org/article/10.3389/neuro.11.017.2009.

[19] A. P. Davison, D. Brüderle, J. Eppler, et al. “PyNN: acommon interface for neuronal network simulators”.In: Front. Neuroinform. 2.11 (2009). doi: 3389/neuro.11.011.2008.

[20] Daniel Brüderle, Mihai A. Petrovici, Bernhard Voggin-ger, et al. “A comprehensive work�ow for general-purpose neural modeling with highly con�gurableneuromorphic hardware systems”. In: Biological Cy-bernetics 104 (4 2011), pp. 263–296. issn: 0340-1200.url: http://dx.doi.org/10.1007/s00422-011-0435-9.

[21] Gaute T. Einevoll, Alain Destexhe, Markus Diesmann,et al. “The Scienti�c Case for Brain Simulations”. In:Neuron 102.4 (May 2019), pp. 735–744. issn: 0896-6273. doi: 10.1016/j.neuron.2019.03.027. url: https://doi.org/10.1016/j.neuron.2019.03.027.

[22] Mattermost, Inc. Mattermost: Open Source, Self-hostedSlack Alternative. url: https://mattermost.com.

[23] Valentina Armenise. “Continuous Delivery with Jenk-ins: Jenkins Solutions to Implement Continuous Deliv-ery”. In: Proceedings of the Third International Work-shop on Release Engineering. RELENG ’15. Florence,Italy: IEEE Press, 2015, pp. 24–27.

[24] Gerrit Code Review. https :/ /www.gerritcodereview.com/. accessed March 11, 2020. 2020.

[25] René Brun, Federico Carminati, and Giuliana GalliCarminati, eds. From the Web to the Grid and Beyond.Springer, Berlin, Heidelberg, 2012.

[26] GNU Lesser General Public License. Version 2.1. FreeSoftware Foundation. url: http : / / www. gnu . org /licenses/gpl.html.

[27] Russ Cox. “Surviving Software Dependencies”. In:Commun. ACM 62.9 (Aug. 2019), pp. 36–43. issn: 0001-0782. doi: 10.1145/3347446. url: https://doi.org/10.1145/3347446.

[28] Gregory M. Kurtzer, Vanessa Sochat, and Michael W.Bauer. “Singularity: Scienti�c containers for mobilityof compute”. In: PLOS ONE 12.5 (May 2017), pp. 1–20.doi: 10.1371/journal.pone.0177459.

[29] Todd Gamblin, Matthew LeGendre, Michael R. Collette,et al. “The Spack Package Manager: Bringing Orderto HPC Software Chaos”. In: Proceedings of the Inter-national Conference for High Performance Computing,Networking, Storage and Analysis. SC ’15. Austin, Texas:ACM, 2015, 40:1–40:12. isbn: 978-1-4503-3723-6. doi:10.1145/2807591.2807623.

[30] B. Gough and R.M. Stallman. An Introduction toGCC: For the GNU Compilers Gcc and G++. Net-work theory manual. Network Theory, 2005. isbn:9780954161798. url: https://books.google.de/books?id=yIGKQAAACAAJ.

[31] Chris Lattner and Vikram Adve. “LLVM: A Compila-tion Framework for Lifelong Program Analysis andTransformation”. In: San Jose, CA, USA, Mar. 2004,pp. 75–88.

[32] S. van der Walt, S. C. Colbert, and G. Varoquaux. “TheNumPy Array: A Structure for E�cient NumericalComputation”. In: Computing in Science Engineering13.2 (Mar. 2011), pp. 22–30. issn: 1558-366X. doi: 10.1109/MCSE.2011.37.

[33] John D. Hunter. “Matplotlib: A 2D Graphics Environ-ment”. In: IEEE Computing in Science and Engineering9.3 (2007), pp. 90–95.

[34] Roman Yakovenko. pygccxml/py++. url: https : / /sourceforge.net/projects/pygccxml.

[35] Boost.Python. Version 1.71.0 Website. http://www.boost.org/doc/libs/1_71_0/libs/python. 2019.

[36] Delta V Software. Remote Call Framework. url: www.deltavsoft.co.

[43] Mario Ruiz, David Sidler, Gustavo Sutter, et al.“Limago: an FPGA-based Open-source 100 GbE TCP/IPStack”. In: Sept. 2019. doi: 10.1109/FPL.2019.00053.

[45] PACKET(7) Linux Programmer’s Manual. Feb. 2020.url: http://man7.org/linux/man-pages/man7/packet.7.html.

[47] Mihai A Petrovici, Bernhard Vogginger, Paul Müller,et al. “Characterization and Compensation of Network-Level Anomalies in Mixed-Signal Neuromorphic Mod-eling Platforms”. In: PLOS ONE 9.10 (2014), e108590.

[52] Boost.Graph. Version 1.71.0 Website. http://www.boost.org/doc/libs/1_71_0/libs/graph. 2019.

[53] Microsoft. TypeScript: JavaScript For Any Scale. url:https://www.typescriptlang.org/.

[54] PixiJS 5. 2019. url: https://www.pixijs.com/.[55] Andy B Yoo, Morris A Jette, and Mark Grondona.

“Slurm: Simple linux utility for resource management”.In: Workshop on Job Scheduling Strategies for ParallelProcessing. Springer. 2003, pp. 44–60.

[56] Katrin Amunts, Christoph Ebell, Je� Muller, et al. “TheHuman Brain Project: Creating a European ResearchInfrastructure to Decode the Human Brain”. In: Neuron92.3 (Nov. 2016), pp. 574–581. issn: 0896-6273. doi:10.1016/j.neuron.2016.10.046. url: https://doi.org/10.1016/j.neuron.2016.10.046.

[57] Human Brain Project. Python client for the HumanBrain Project Neuromorphic Computing Platform. url:https : / / github . com / HumanBrainProject / hbp -neuromorphic-client.

[58] Graphite Project. Carbon. url: https://github.com/graphite-project/carbon.

[59] M. Massie, B. Li, B. Nicholes, et al. Monitoring withGanglia. Oreilly and Associate Series. O’Reilly Media,Incorporated, 2012. isbn: 9781449329709. url: http://books.google.de/books?id=w4LLpXeVCbkC.

[60] R. Gerhards. The Syslog Protocol. RFC 5424. RFC Editor,Oct. 2009. url: https://www.rfc-editor.org/rfc/rfc5424.txt.

[61] elastic. Logstash: Collect, Parse, Transform Logs. url:https://www.elastic.co/logstash.

[62] elastic. Elasticsearch: The O�cial Distributed Search& Analytics Engine. url: https : / / www. elastic . co /elasticsearch.

[63] Grafana Labs. Grafana: The open observability platform.url: https://grafana.com.

[64] elastic. Kibana: Explore, Visualize, Discover Data. url:https://www.elastic.co/kibana.

[66] Akos F. Kungl, Sebastian Schmitt, Johann Klähn,et al. “Accelerated Physical Emulation of BayesianInference in Spiking Neural Networks”. In: Frontiers inNeuroscience 13 (2019), p. 1201. issn: 1662-453X. doi:10.3389/fnins.2019.01201. url: https://www.frontiersin.org/article/10.3389/fnins.2019.01201.

[67] Julian Göltz, Andreas Baumbach, Sebastian Billaudelle,et al. Fast and deep neuromorphic learning with time-to-�rst-spike coding. 2019. eprint: arXiv:1912.11443.

[68] Johann Klähn. genpybind software v0.2.0. 2020. doi:10 . 5281 / zenodo . 372674. url: https : / / github . com /kljohann/genpybind.

[69] Eric Müller, Christian Mauch, Philipp Spilger, et al.“Extending BrainScaleS OS for BrainScaleS-2”. In: arXivpreprint (Mar. 2020). url: TODO.

Own Software[37] Sebastian Jeltsch. rant. url: https://github.com/ignatz/

rant.[38] Electronic Visions(s), Heidelberg University. pythonic.

url: https://github.com/electronicvisions/pythonic.[39] Sebastian Jeltsch. bitter. url: https://github.com/ignatz/

bitter.[40] Electronic Visions(s), Heidelberg University. halco. url:

https://github.com/electronicvisions/halco.

[41] Electronic Visions(s), Heidelberg University. halbe. url:https://github.com/electronicvisions/halbe.

[42] Electronic Visions(s), Heidelberg University. sthal. url:https://github.com/electronicvisions/sthal.

[44] Eric Müller, Moritz Schilling, and Christian Mauch.HostARQ Slow Control Transport Protocol. Apr. 2018.url: https://github.com/electronicvisions/sctrltp.

[46] UHEI, TUD. ESS. url: https : / / github . com /electronicvisions/systemsim-stage2.

[48] Electronic Visions(s), Heidelberg University. redman.url: https://github.com/electronicvisions/redman.

[49] Electronic Visions(s), Heidelberg University. calibtic.url: https://github.com/electronicvisions/calibtic.

[50] Electronic Visions(s), Heidelberg University. pyhmf.url: https://github.com/electronicvisions/pyhmf.

[51] Electronic Visions(s), Heidelberg University. marocco.url: https://github.com/electronicvisions/marocco.

[65] Electronic Visions(s), Heidelberg University. cake. url:https://github.com/electronicvisions/cake.