Project #5

Purpose:

Software Collaboration Federation (SCF) is a collection of clients and servers and associated software designed to support activities of a software development team, e.g.:

Creating and publishing plans for development. This includes writing concept documents, creating and editing work packages, scheduling work packages, and allocating resources to work packages.
Writing and publishing specification and design documents.
Preparing new source code packages
Acquiring existing source code packages for reuse
Building execution images and loadable libraries
Executing tests: unit tests, integration tests, regression tests, performance tests, stress tests, and acceptance tests.
Deploying software executables and documentation.
Reporting progress, key events, and failures.

In the Fall 2012 semester we developed an architecture for a Virtual Display System (VDS). The system in which it is embedded is shown in Figure 1., below. It is intended to support collaboration for large software development teams. One of its primary features - the one we focused on - is a large display that serves as a visual interface into the design and implementation of complex software systems. To understand how VDS functions you should review the preliminary architectural document provided to the Fall 2012 class, Preliminary Architecture for the Virtual Display System.

In this project we will explore how to use the noSQL database we built in Projects #2 and #4 to manage recording and reporting of important SCF data and events. We will assume that SCF is intended to support development teams comprised of several hundred developers divided into groups at different locations throughout the world.

Background:

The SCF is comprised of a federation of clients and servers¹:

Collaboration Server: supports project management, storing work package details and providing collaboration tools.
Repository Server: Stores source code and documents using a configuration control system and provides exploratory tools based on metadata associated with each stored item.
Build Server²: builds and caches execution images and libraries as needed for test and deployment.
Test Server: loads and executes test images provided by the build server. The test server also provides deployment activities.
Virtual Display Server: provides a sophisticated interface for publishing management information and collaboration activities that involve viewing source code, documents, diagrams, sketches, and webcams to enhance personal and team interactions. This is a prime vehicle for collaboration between remote teams.
Development Clients: provide user access to the many facilities of VDS including check-in and check-out of code and documents, creation and scheduling tests, and viewing results³.
Virtual Display Client: a specialized client that focuses on managing displays of documents, code, web cam views, and various charts and tables that describe progress and interactions between development groups.

Figure 1. Software Collaboration Federation (SCF)

The Storage Management Subsystem (SMS) provides a core SCF service⁴ and is composed of Data and Event Managers hosted on each of the clients and servers in the VDS Federation and a Data and Event Coordinator that supervises the operation of all of the individual Data and Event Managers.

Storage Management Subsystem Architecture:

The Storage Management System provides a number of services including storing and managing:

Routing lists for communication of messages and notifications
Layout information for VDS: where are the windows, what content do they display
Metadata for source code and documents in the repository
Check-in/out status for code and documents in the repository
Test configurations and results
Lists of services and servers available to clients
Templates for displays and messages

In your Architecture Document you should summarize the SCF structure and behaviors, and then focus in detail on the Storage Management Subsystem. For that you should:

Define its structure and the internal interactions⁵ between elements of the Storage Management Subsystem and external interactions with other services and components of the Software Collaboration Federation.
Discuss behaviors that are common to all the Data Storage Managers.
Discuss behaviors that are unique to each of the Managers and how the implementation will merge the unique and common operations.
Discuss behaviors of the Data Storage Coordinator, how it will be hosted, and its interactions with the rest of the Subsystem as well as with the other parts of SCF.
Enumerate all of the data structures that SMS will have to handle, e.g., their structure, purpose, performance issues, and any other constraints that occur to you.
Discuss how storing and managing new data configurations can be added to an existing SMS.
Define all of the events that will be handled by SMS:
- Origin of the event.
- Nature of the event, e.g., business as usual events - checkin, successful build completion, successful test and errors or anomolous behavior.
- Notifications that result and their destinations.
- Event information that will be stored, that will be transmitted, and the lifetime of the information.
- Priority of messages and mechanisms for expediting the transmission of high priority messages.
Discuss how new events can be added and managed in an existing SMS.
Discuss critical issues: complexity, security, performance, integration of SMS with the SCF communication service and with its other parts if you think there should be interactions with parts other than the communication service.
How will SMS support giving users deep insight into the operations of SCF via: message content, message configuration, orchestrating the time of delivery of messages, displaying stored data content, and showing the data traffic and evolution of stored data?

Prototypes:

In order to support the many decisions required to define the architecture of a large interesting software construct like SMS, an architect will build a set of prototypes to understand how things should work, explore performance issues, and decide on specific structures and patterns of communication.

A prototype should be:

As small as is possible while answering the questions that caused its construction.
Simple enough that developers can understand it and use it as the basis for a more complete solution.
Provide output information that contains convincing answers about the questions.
Documented as an integral part of the architectural document(s).

You will be required, below, to implement at least two prototypes for Project #5.

Requirements:

Your Storage Management Subsystem (SMS) architectural document:

Shall be written using Microsoft Word or an equivalent office suite that can generate documents in word or PDF format.
Shall use diagrams created with Microsoft Visio or an equivalent tool that can generate diagrams embedded in your document. These diagrams should be delivered in Visio, PDF, or PNG formats.
Each diagram shall be accompanied by text that describes:
- Why this diagram is here.
- What the diagram tells us.
- Conclusions we should draw from its presentation.
Shall, at the beginning provide an Executive Summary that very briefly summarizes the main parts of your architecture, especially its users and uses, parts, and critical issues. The summary does not list the things you did to develop the document.
Shall briefly discuss the goals of the SCF and discuss in more detail the goals of the SMS.
Shall briefly discuss the uses and users of the SCF and discuss in more detail the uses and users of the SMS. This discussion should consider the impact of uses on the design of SMS, explore uses beyond those specific to SCF⁵ and decide if any of them need to be addressed in the development of SMS, e.g.: provide for future extensions and additions.
Shall briefly discuss the structure of the Software Collaboration Federation and then discuss in detail the structure of the Storage Management Subsystem, its data, and its events.
Shall discuss the responsibilities of each of the parts identified above and interactions between them.
Shall discuss issues that are critical for the development of SMS and propose means to address each of the issues deemed to be important for SMS.
Shall, at an appropriate place in the document, discuss each of at least two prototypes you've developed. You will develop the prototypes:
- Event Logger:
  The event logger records the type of event, the date-time of its occurance, and its origin. The logger has its own process and is accessed by code in other processes through the SCF communication service which you will represent with a WCF service which is part of your prototype. The purpose of this prototype is to estimate performance: throughput and latency of logging. Note that a lot of this prototype will be taken directly from your implementation of Project #4. You are required to use .Net Tasks as part of your implementation.
  If you implement this prototype so that it can be conveniently used in process as well as between processes this will be counted as an additional prototype. Note that that does not replace either of the two required prototypes described here.
- Critical Issue Prototype(s):
  This prototype is developed to help you answer questions and provide solutions to one or more of your critical issues. What you do and how you do it will be left to you. You will find me reluctant to discuss this in any detail.

Clients and servers are usually identified by machine boundaries but that is not always the case. We may implement more than one server as virtual machines on a single physical machine. Also, we may have more than one physical machine that implements a server. For example, toward the end of a large project testing becomes a very intense large-scale activity and might be carried out on a Test Server Farm.
The Build Server will probably share the same machine as the Repository and so is not shown in Figure 1.
These clients are often processes that run on a developer's machine. SCF clients are not expected to provide compilers and editors and other build tools. Those already exist and would be a waste of resources to re-implement. So a developer's machine would have both the tools and the SCF client software.
One would expect a system like SCF to have several services around which its functionality is configured, e.g., services for communication, security, storage, display, ...
The SMS will contain many parts that interconnect with SCF and that handle it's internal operations, e.g., data storage, query handling, persistance, message handling, and internal error handling.
If we don't look to future expansion of capabilities we may find that after a big expense and effort to build the (sub)system we end up with an inflexible body of software with no growth potential. That's definately not what an architect is paid to create.

What you need to know:

In order to successfully meet these requirements you will need to:

Understand the purpose of an architecture document, which is an elaboration of the Operational Concept Documents you developed in Projects #1 and #3.
Understand the details of the design and implementation of Projects #2 and #4.

Things for which you lose points:

Weak presentation of your discussions, e.g.: inadequate detail and inaccuracies in detail.
Formatting issues: your text and diagrams should convey their information in clear, simple fashion. Formatting should be consistent. If your diagrams and text are inconsistent in format, structure, and tone, it becomes clear that you did a lot of cut and paste from the web and class documents rather than thinking through issues and their presentation yourself.

Things for which you gain points:

I value highly your presentation of original ideas and discussion of original structures.
Your content should provide a broad treatment of a lot of the system parts and behaviors with modest detail and much more detail for those parts you think are particularly interesting and/or important. The broad treatment teaches your reader about the system and its context. The detailed treatment tells the reader about the important things you uncovered or developed.
Feel free to bend the preliminary architecture presented above in ways you think are useful and intersting.