spyGUI: A User Interface to the Online Monitoring of the Silicon Vertex Tracker in CDF II

Marco Rescigno, Subir Sarkar

Abstract

spyGUI is a user interface to the Online Monitoring system (SPYMON) for the CDF Silicon Vertex Tracker (SVT). spyGUI displays SVT hardware and analysis errors in detail, decodes raw data into physics quantities, provides interactive plotting facility, works as an Online Beam Position Monitor, works as a Histogram Producer and Viewer, configures SPYMON analysis options, works as an SVT Database Browser, and performs simple error recovery. In this document we'll discuss the architecture and design of spyGUI and describe the components of the system in detail.

1. Introduction

spyGUI is a Java based graphical user interface to the Online Monitoring and analysis framework (SPYMON) for the CDF Silicon Vertex Tracker which is built to study the integrity and performance of SVT[1,2] in real time. The goals of SPYMON range from very simple statistical analysis at single board level to more complex ones such as multi-board event building and multi-crate checks; from simple error reporting to expert level diagnostics and possible error recovery. In order to achieve such diverse goals, SPYMON exploits several diagnostic devices integrated in the SVT design in order to provide several ways of verifying the integrity of data-flow through the entire system. SPYMON also calculates the Beam Position parameters in real time, from the measured track parameters. The entire SPYMON system comprises of a number of tools as shown in Figure 1 and described briefly in the following. For a detailed discussion of SPYMON refer to [3].

• SPYMON proper runs on all the 8 SVT crate CPUs, generates SVT_FREEZE, collects and analyses Spy Buffer data, reads board error registers and subsequently publishes the following three kinds of SmartSockets messages,
  • histograms of physics (Roads, Tracks etc.) and diagnostic (board and analysis error flags) quantities
  • SVT status messages containing error registers and status words for each of the boards and statistics about individual error register flags
  • Spy Buffer dumps, i.e. formatted dump of part/all of the Spy Buffer data.

  In addition to that, whenever the beam finding algorithm finds a new fit of beam position (in crate b0svt06), a special message is published with fit results, for the 6 different SVX barrels. For each barrel, the same results are published for each of the 12 phi wedges also. See Online Beam position monitor page[4] for detail.

• SpyManager is a Unix process that runs continuously as a book-keeper of SPYMON results. It
  • collects Spy Buffer dump messages and writes them onto disk for further analysis
  • collects SVT status messages and updates the SVT Crates Status web page which links to individual crate status pages on the B0 cluster
  • collects Beam position updates and sends them to ACNET using XML-RPC. The beam parameters from ACNET can be found in SVT Parameters from ACNET.

  The SpyManager can, in principle, be made part of a run (by including in the partition the ``pseudo-crate'' SVT\_MONITOR\_OO) though little is gained by doing so for the moment except for the fact that this way it will be able to communicate with the Run Control for special error condition/messages and act as an SVT error logger.

• SVTSPYMON, a batch Unix process that receives SVT histogram messages from the SVT crates and converts them into Root histograms. The histograms are then shipped to the standard DisplayServer and can be browsed using the standard HistoDisplayMain program. SVTSPYMON is described in detail in [5].

• spyGUI, a Workstation process which collects informations from the SVT Crate processes and displays them in real time after subsequent analysis. spyGUI has been designed to perform all the tasks using a single application that are individually carried out by different components of the entire SVT Online Monitoring and Analysis system. In particular, spyGUI is meant to,

  • display SVT hardware and analysis errors in detail,
  • decode raw data into physics quantities,
  • work as an Online Beam Position Monitor,
  • configure Spy Buffer analysis options,
  • work as a Histogram Producer and Viewer,
  • work as an SVT Database Browser,
  • perform simple error recovery.

  spyGUI will be discussed in detail in this note.

• svtGUI, a graphical user interface for standalone single board diagnostic studies. See [6] for detail.

Figure 1: SPYMON components and interaction.

spyGUI is a SmartSockets client which communicates with the SVT crate processes (and optionally with the Spy Manager). The left part of Figure 2 shows the source of spyGUI data whereas the right side represents schematically the main system blocks of the user interface and their interrelation. In particular, the arrows indicate direction of information flow and upto what level information can be browsed starting from the main window. The same figure also shows the package structure of spyGUI code.

Figure 2: spyGUI components, their interrelation and package hierarchy. The direction of flow of information and consequently the depth upto which information can be accessed are also indicated.

2. Design and Implementation

2.1 Tools and Usage

In this section we'll discuss about the existing SPYMON tools and the scope of spyGUI as an integrated analysis framework. We'll describe as well the different ways spyGUI can be used and their respective merits and demerits. The various tasks SPYMON was originally designed to carry out are listed in the first column of Table 1. The second column lists the individual tools that perform the corresponding tasks. One notices that, although, SPYMON is quite rich in functionalities, the tools are scattered and require maintenance effort in order to synchronise the tools against any modification in the data structure. spyGUI integrates these scattered tools in a single application and provides the missing functionalities that makes SPYMON more powerful and easier to configure. It might be noted that spyGUI originated to provide a simple configuration tool for the SPYMON system. There are certain offline utilities which perform spy buffer analysis.

spyGUI can be used in two logically distinct ways as described below.

• Using a single entry point where only one SmartSockets client performs all the communications and analysis,

          > setup svtmon -d
          > spyGUI 
       

  This was the original design; although it makes usage of spyGUI straight-forward, it also complicates navigation which is too many levels deep and makes it rather heavy from the point of view of memory usage.
• Independent SmartSockets clients for individual components,

         > setup svtmon -d
         > java config.SendCommandFrame
         > java config.SvtConfigurator
         > java config.SvtStatusViewer
         > java config.SvtBoardErrorViewer
         > java config.SpyErrorViewer
         > java config.svt.SpyBufferViewer
         > java config.beam.BeamMonitor
         > java config.hist.HistogramViewer
         > java config.db.SvtDbBrowser
       

  In the new scheme memory usage remains under control; more importantly, since one can freely use only the tools that are needed, browsing information becomes much simpler. Note that, a number of SmartSockets clients will be required in this scheme in the real situation. Fortunately, not all the clients need to be run at a time. Clearly, the new scheme is the preferred and recommended way to use spyGUI.

2.2 Architecture

spyGUI is extremely dynamic in nature. Most of the graphic components and many important data structures are created at Runtime. As an application starts, it only sets up the required SmartSockets connections and waits for messages to be published by the SVT Crate processes. On receipt of messages from various crates, information is decoded, and subsequently graphic components are instantiated and data structures filled. This is true for the overall spyGUI as well as for all the individual components. This way the graphical representation of spyGUI is always validated.

spyGUI was not designed from ground up; the project followed an incremental development strategy. As a result, code had to be often retrofitted in order to implement certain design aspects which were found to be useful along the way. This include preferring interface and composition based development over inheritance, profiting from well known design patterns like Singletons, Observables and Observers etc. We'll briefly discuss below about some design related features which made development of spyGUI easier and increased modularity and pluggability of different components inside the framework.

Figure 3 shows the inheritance hierarchy of the graphics windows. DataFrame, which inherits from JFrame, the top level class representing a Frame in Java, implements a lot of common behaviour like, (1) Menu Bar (2) Tool Bar (3) Status Bar (4) Embedded Text Panel (5) Window closing strategy (6) Online Help etc. It also implements the a few basic Runtime configuration for windows. This provides a consistent look and feel for all the visible components.

Figure 3: Inheritance hierarchy of the graphic widgets. spyGUI and the individual applications are marked with coloured filled boxes.

The physics classes are needed by a number of components of spyGUI, for analysis as well as for display. Figure 4 shows how the physics classes are formed from raw Spy buffer data from an Svt crate to achieve required flexibility. Each raw event directly translates to a generic SvtEvent, an abstract class, which consists of an object that describes event data for a board spy buffer (SvtObject, an interface) and an End Event word. A number of such events form SvtEvents, an abstract class which is a collection of events. The concrete classes specific to a particular board buffer follow from the abstract classes and interfaces.

Figure 4: Hierarchical structure of the Physics classes which provides flexibility to spyGUI analysis. The figure also show how the raw data are stored in hierarchical structures, from a single SvtBufferData to an SvtCrateData.

The following Figures 5,6,7 show simplified UML class diagrams for the abstract classes and interfaces and how they are extended and implemented in concrete classes.

Figure 5: Implementation of the interface, \emph{SvtObject} which provides the concrete classes required pluggability in various places, i.e Spy Buffer analysis, interactive histogramming and various display classes.

Figure 6: Inheritance hierarchy of the abstract class, SvtEvent.

Figure 7: Implementation of the interface, SvtEvents. The concrete classes implements additional interfaces (e.g SvtHistogrammer) for greater flexibility

3. Description

In its present configuration, spyGUI runs outside the DAQ partition. It knows about the various Run Control state transitions from the crate processes. The application uses multiple threads of execution. The main thread manages all the user interactions, like navigating though error informations, histograms and modifying analysis configurations etc. A separate thread manages SmartSockets communications and accordingly updates informations on the client side.

Figure 8 shows a screen-shot of the main spyGUI window.

Figure 8: spyGUI main window. The upper half of the window changes color as SmartSockets connection is established and messages are received from crates. The bottom part contains a number of notebook pages to show Beam Positions, raw Spy Buffer dump etc.

At startup, one can choose the subject areas the application should subscribe to from a dialog box as shown in Figure 9.

Figure 9: Subscription dialog. spyGUI can choose to initially subscribe to a certain type of SmartSockets messages from the list.

Figure 10: Subscription menu. spyGUI can dynamically subscribe to/unsubscribe from a certain type of SmartSockets message.

The bottom part of the main window contains a number of display windows to log various activities. The Beam Position tab displays updated Beam Positions in a table as described in a later section.

3.1 SVT Status Display

The window shown in Figure 11 roughly emulates the SVT hardware crate/board configuration (without cable connections). Standard colours for different elements, i.e board panels are used to simplify board identification without explicitly labeling them. For more information on board colour codes consult the images stored in the SVT page.

Figure 11: SVT status window, gives an up-todate summary of SVT status. Individual board panels as well as the buttons with a label above the Crate panels can be picked to access more information. Separate indicators for all the 8 crates notify if crates are sending data (online).

Boards which do not have Spy Buffers are passive elements. The crate CPU panel is a button which shows a summary of status of all the Spy Buffers which belong to that crate. Alternatively, one can click on the buttons which show the (logical) crate names to display the same informations.

Error Colour Codes:

• Red - Error has occurred in that element
• Green - Running alright as Spy Monitoring finds it
• Yellow - Element not available for monitoring

This window should summarise all the relevant informations about the functioning of SVT and hence may be open all through. The LEDs on each board represent the following as explained graphically in Figure 12.

Figure 12: Explanation of status error indicated in each Board panel.

• Top Panel
  • EE Event related error indicator. Error is ON if any of the Spy Buffers in that board has an EE error.
  • Spy Buffer related error indicator. Like the one above, but refers to the error related to Spy Buffer analysis, like simulation etc.
  • Output Hold Error indicator
• Bottom Panel
  • Board-in-Run/Test mode indicator
  • CDF/SVT error indicator, an OR of the two
  • Board error register indicator. It is an OR of all the board error registers.

The LEDs are buttons as well and can be selected to display details of the occurrence of an error. By default a new frame is opened following a button click which contains error informations for the whole system. One can, alternatively, check Show Errors on Separate Windows to views informations only for a single element.

3.2 Board Error Display

One can navigate through summary informations about different SVT Errors as described below briefly using a single window as shown in Figure 13,

• End event Errors as registered by the Spy Buffers
• Spy Buffer analysis (Simulation) related errors
• CDF/SVT Errors
• Test/Run Mode and board output hold flags
• Board Error Registers

Figure 13: SVT Board error details. Information for all the Boards can be browsed from a single window.

The LEDs indicate occurrence of error during the most recent iteration. For Board Error registers total number of occurrences of error of each kind since the last reset is also shown.

3.3 Analysis Error Display

As shown in Figure 14, Spy Buffer Error Window displays a summary informations of global (End Event) errors for all the Spy Buffers of all the boards present in a crate. If more than one crates send error messages, the crate panels are added as separate pages. One may click on individual buttons to navigate through relevant informations on the errors in a separate window.

Figure 14: Spy Buffer Analysis Error Summary Window. We use the 'SlideShow' option here such that the controls cycles through all the crates (implemented as notebook pages) with a delay. Error indicator icons are associated with buttons which can be used to pick a certain buffer for detailed information (as in Figure 21).

Tool-tips are provided extensively as the buttons are small and the names are cryptic. When the mouse enters/rests on a button a brief description is displayed on the status area as well.

3.4 Configuration

spyGUI can be used to configure Spy Monitoring analysis at various levels.

• User level configuration; this can be divided in 2 parts
  • only the Workstation side settings to be changed, but reading and analysis of data within the crates will not be influenced.
  • configure analysis in order to change the amount of information that will be published by the crate processes.
• Expert level configuration tool, which would allow one to modify the buffer reading, simulation, histogram filling options etc. within the crate processes. We rarely felt any need for more advanced options, like dynamically booking and filling new histograms etc.

3.4.1 Sending Commands to Crate Processes

We can send application specific commands to the selected destination(s) using the following predefined commands as shown in Figure 15,

• pause, resume, stop the SPYMON process in the selected crate(s); reset, configure certain options, trigger an action.
• sendStatus; publish crate/board status information
• sendHist; publish histograms. resetHist; reset Histogram content
• resetError; reset Board Register Errors

Figure 15: SVT Command Sending Tool. Simple commands, a brief configuration message with predefined length, configuration message of arbitrary length using files can be sent using this utility.

The Stop command can be used to stop the consumer processes as well. In future we may add a free text input area used to send the command name.

One can also send configuration messages from a file, either by sending

• simply the fully qualified filename the content which will be read by the receiver, or
• the file contents as strings which will be decoded directly by the receiver

Sending configuration from file could be useful when the same configuration properties are altered time and again.

3.4.2 Spy Buffer Analysis Configuration

User as well as expert level analysis configurations can be performed using this tool. A single message may be created, either for one crate at a time or a common one valid for all the crates and published via SmartSockets.

Figure 16: SVT Crate Configuration. Detailed configuration of the SPYMON analysis can be performed using this tool. The default configuration is loaded from file at startup which is modified and sent as SmartSockets messages. The modification can be saved as new default.

The configuration options are described in detail below,

• Mode of monitoring,
  • Wait for Run\_Control partition to become available
  • Do routine tasks even when the partition does not come into being.
• Run Mode,
  • Routine pole and reading of Spy Buffers even if there is no freeze from outside. This option has to be elaborated in future to make it more flexible.
• Spy Buffer Analysis options
  • Analyse/Skip the buffer under consideration
  • Read Spy Buffers
  • Dump to disk
  • Fill and publish histograms
  • Perform board/crate based simulation
• For a selected buffer,
  • Frequency of buffer reading
  • Number of words to be read
  • Maximum number of words to be dumped
• Type of histograms to be booked/filled/published
  • Event Error histograms
  • Standard monitoring histograms
  • Histograms of physics quantities of interest
  • Histograms to be consulted by SVT Expert

3.5 Database Browser

A tiny SVT Database Browser has been added which displays the SVT hardware configuration as stored in the Database.

Figure 17: SVT Database Browser.

By default the window displays the active configuration. The 'View' panel displays

• the mapset/hwset filenames with their respective CRCs and timestamps.
• the names of the Pattern, SSMap, fcon files for each of the 12 wedges. For fcon, only one file (barrel 0) for each wedge is displayed.

One should also be able to display SVT configuration for an arbitrary Run Number for which the configuration exists. The 'Reload Active' once again restores the active configuration.

The 'Operation' panel allows one to

• display one file at a time. A separate window will be opened for each file.
• save selected file(s) to disk
• save all the files shown in the list to disk
• validate the selected file(s), i.e, calculates checksum (CRC) of the file and compares with the
• value that is stamped in the file itself.

One can use a popup menu attached to the list in the 'Operations' panel to

• Select Multiple element selection mode
• Clear existing selection
• Display names of all the mapset files present in DB
• Display names of all the hwset file present in DB
• Return to the active configuration

Implementation of validating a DB entry validation (Calculating CRC) depends on Native C code.

3.6 Histogramming

A Histogram producer and viewer that is very close to the Root based Histogram producer (SVTSPYMON) in capabilities has been included in spyGUI. There are occasions when SVTSPYMON stops working for brief periods intermittently for technical reasons like Root version mismatch etc.; in such cases the Java based Histogram Producer we are discussing here may come handy. Figure 18 shows a screen-shot of the viewer window. The viewer is built upon Java Analysis Studio (JAS) developed by Tony Johnson[7].

Figure 18: Histogram Browser window.

Histograms are created/updated from histogram messages published by the SVT Crates. At present only 1- and 2-D histograms are supported. For two dimensional histograms only binned informations are available. This is enough for us since this functionality covers all kinds of histograms that are published by SVT crates as messages.

1. a list view which displays the histogram list as a tree structure
2. a plot canvas.

One can divide the canvas into a number of zones as shown in Figure 18. Individual plotting panels are active and one can click the right button on various objects (data, axis, statistics box etc.) to popup self-explained menus which can be used to customise almost all the histogram attributes like, line, background and fill colours, statistics elements, error bars, axis titles, axis ranges etc. The histograms being displayed can also be printed in this way. However, if more than one histograms are displayed, saving/printing may be cumbersome. More work is required to implement a better printing method.

The application Toolbar contains the following buttons,

• Save text from the message area in a file
• Configure (not yet implemented)
• Plot the selected histogram
• Plot the previous histogram
• Plot the next histogram
• Clear canvas
• Change number of zones in the canvas
• Dump ASCII content of the selected histogram in the message area, mainly for debugging purposes
• Open a help window (button at the right corner)

3.7 Beam Position Monitor

spyGUI displays the results of the Beam Position measurement online in a tabular format in the main application window as shown in Figure 19.

Figure 19: Beam Positions for the 6 SVX barrels.

A new application, BeamMonitor(JAS version 2.2.5 or higher is required for BeamMonitor), was added for more information which is shown in Figure 20. The BeamMonitor can be used as a separate SmartSockets clients in order to display Beam Position values for the 6 SVX barrels as well as for each of the 12 SVX phi sectors (wedges) for all the 6 barrels. The Beam Positions are also presented graphically in the lower half of the BeamMonitor window.

Figure 20: Beam Positions for the 6 SVX barrels as well as for each of the 12 phi wedges for the 6 barrels.

4 Analysis

One can browse through all the spy buffer data published during the most recent SPYMON iteration from a single window in Real time, limited only by the amount of information published by the crate processes. The buffers are automatically updated each time new data are received; however automatic update can be switched off/on at Runtime from the 'Option Menu'. One can choose to receive data only from a particular crate as well. The same tool can be used to analyse Spy dump files offline, e.g from earlier stores by loading them from the 'File Menu'. At present only one file can be analysed at a time.

The top-level window consists of a tree structure on the left which lists all the spy buffers which are read and analysed by SPYMON crate processes and marks the ones that receive data presently and a notebook with the following three pages,

• Global Error (EE) status(see Figure 21), which consists of
  
  
  Figure 21: Global Error (EE) status of the individual Spy Buffers.
  • Spy Buffer status window,
    • Spy pointer value
    • Freeze status bit
    • Pointer overflow bit
  • Status of the 8 individual EE Error bits during the most recent iteration
  • EE error informations in more detail in tabular form. The table (at the bottom of Global Error window) displays
    • Number of times EE errors have occurred and the \% fraction of occurrence of EE errors in the latest buffer.
    • Total cumulative number of times EE errors have occurred and the \% fraction of occurrence of EE errors.
• Spy Buffer data display which shows data Board specific format in the top panel as well as in raw hexadecimal format in the bottom panel. Options exist which allow one to view the End of Event (EE) words only as opposed to all the words. A statusbar at the bottom shows the number of words being displayed and the options that were chosen.
  
  
  Figure 22: Individual Spy Buffer data window. Both the interpreted and raw data are shown simultaneously.

• Plotting Widget where physics quantities are histogrammed. Useful quantities are filled and kept in memory, either cumulatively or on-an-iteration basis. Figure 23 shows a number of predefined histograms for the XTFA board spy buffer.
  
  
  Figure 23: Predefined histograms for a single buffer. Histograms can accumulate events over a large period or can use events from a single iteration.

  A plotting widget for dynamic histogramming as shown in Figure 24 as well has been included as well. A Jython interpreter has been embedded in spyGUI for dynamic histogramming. This is still very experimental and not yet optimised for efficiency.

  
  
  Figure 24: Interactive Plotting widget. Only predefined variables can be used. 1D histograms and 2D scatter plots are supported. The plotting widget is not yet optimised for efficiency.

4.1 Svt Word Decoder

Figure 25: SVT Word Decoder, transforms data from raw hexadecimal format to interpreted format.

A tiny tool has been integrated in spyGUI to decode SVT words interactively from raw Hexadecimal format to relevant physics quantities. The window is divided into two parts. The upper part consists of an input text area where raw hexadecimal data should either be typed, copied from another Java window or imported from file(s) and a panel of options to choose the board format from. The bottom part displays the formatted output according to the board that will be selected. Presently, the decoder can transform,

1. End Event Words
2. Hit words
3. Roads
4. Road-Hit Packets
5. Tracks words into the corresponding board/physics format. A simple example is shown in Figure 25.

The decoder output format is exactly the same as the one found in the Spy Buffer Viewer window; the same code was used for both. The Svt Word Decoder is a very handy tool for debugging.

5 Conclusion

References

1. S. Belforte et. al, IEEE Trans. Buck. Sci. 46, 933(1999) W. Ashmanskas et. al, Nucl. Instr. Meth. A447, 218 (2002)
2. R. Blair et. al [CDF Collaboration], MILAN-PUB-96-390-E
3. M. Rescigno, S. Sarkar and L. Zanello, CDF Note CDF/PUB/ONLINE/PUBLIC/7188, 2004.
4. http://www-cdfonline.fnal.gov/svt/beam
5. http://www-cdfonline.fnal.gov/svt
6. A. Barchiesi and S. Sarkar, CDF Note CDF/PUB/ONLINE/PUBLIC/6504.
7. http://jas.freehep.org