Project #5

Purpose:

Software Collaboration Federation (SCF) is a collection of clients and servers and associated software designed to support activities of a software development team, e.g.:

Creating and publishing plans for development. This includes writing concept documents, creating and editing work packages, scheduling work packages, and allocating resources to work packages.
Writing and publishing specification and design documents.
Preparing new source code packages
Acquiring existing source code packages for reuse
Building execution images and loadable libraries
Executing tests: unit tests, integration tests, regression tests, performance tests, stress tests, and acceptance tests.
Deploying software executables and documentation.
Reporting progress, key events, and failures.

In the Fall 2012 semester we developed an architecture for a Virtual Display System (VDS). The system in which it is embedded is shown in Figure 1., below. It is intended to support collaboration for large software development teams. One of its primary features - the one we focused on - is a large display that serves as a visual interface into the design and implementation of complex software systems. To understand how VDS functions you should review the preliminary architectural document provided to the Fall 2012 class, Preliminary Architecture for the Virtual Display System.

In this project we will explore how to structure the SCF, with emphasis on supporting continuous software test and integration. You should assume that the Test Harness server provides the functionality we developed in Project #4. We will assume that SCF is intended to support development teams comprised of several hundred developers divided into groups at different locations throughout the world.

Background:

The SCF is comprised of a federation of clients and servers¹:

Collaboration Server: supports project management, storing work package details and providing collaboration tools.
Repository Server: Stores source code and documents using a configuration control system and provides exploratory tools based on metadata associated with each stored item.
Build Server²: builds and caches execution images and libraries as needed for test and deployment.
Test Harness Server: loads and executes test images provided by the build server. The test server also provides deployment activities.
Virtual Display Server: provides a sophisticated interface for publishing management information and collaboration activities that involve viewing source code, documents, diagrams, sketches, and webcams to enhance personal and team interactions. This is a prime vehicle for collaboration between remote teams.
Development Clients: provide user access to the many facilities of SCF including check-in and check-out of code and documents, creation and scheduling tests, and viewing results³.
Virtual Display Client: a specialized client that focuses on managing displays of documents, code, web cam views, and various charts and tables that describe progress and interactions between development groups.

Figure 1. Software Collaboration Federation (SCF)

SCF will use a Storage Management Subsystem (SMS) that provides a core SCF service⁴ and is composed of Data and Event Managers hosted on each of the clients and servers in the SCF Federation and a Data and Event Coordinator that supervises the operation of all of the individual Data and Event Managers.

The Storage Management System provides a number of services including storing and managing:

Routing lists for communication of messages and notifications
Layout information for VDS: where are the windows, what content do they display
Metadata for source code and documents in the repository
Check-in/out status for code and documents in the repository
Test configurations and results
Lists of services and servers available to clients
Templates for displays and messages

Software Collaboration Federation Architecture:

In your Architecture Document you should summarize the SCF structure and behaviors, and then focus in detail on support for Continuous Test and Integration (CTAI) using the Test Harness Server and the Storage Management Subsystem. For that you should:

Define the structure and the internal interactions⁵ between elements of the Test Harness and the Storage Management Subsystem and external interactions with other services and components of the SCF.
Discuss behaviors that are common to all the Servers⁶.
Discuss behaviors that are unique to each of the SCF servers and how the implementation will merge the unique and common operations.
Discuss behaviors of the Data Storage Coordinator, how it will be hosted, and its interactions with the rest of the Subsystem as well as with the other parts of SCF. Please emphasize its role in the interactions between Test Harness Server and the SCF Clients.
Enumerate all of the data structures that SCF servers will have to handle, e.g., their structure, purpose, performance issues, and any other constraints that occur to you.
Discuss how storing and managing new data configurations can be added to an existing SMS.
Define all of the events that will be handled by SMS in its role supporting SCF operations:
- Origin of the event.
- Nature of the event, e.g., business as usual events - checkin, successful build completion, successful test and errors or anomolous behavior.
- Notifications that result and their destinations.
- Event information that will be stored, that will be transmitted, and the lifetime of the information.
- Priority of messages and mechanisms for expediting the transmission of high priority messages.
Discuss how new events can be added and managed in an existing SMS.
Discuss critical issues: complexity, security, performance, integration of SMS with the SCF communication service and with its other parts if you think there should be interactions with parts other than the communication service.
How will SMS support giving users deep insight into the operations of SCF via: message content, message configuration, orchestrating the time of delivery of messages, displaying stored data content, and showing the data traffic and evolution of stored data?

Prototypes:

In order to support the many decisions required to define the architecture of a large interesting software construct like SMS, an architect will build a set of prototypes to understand how things should work, explore performance issues, and decide on specific structures and patterns of communication.

A prototype should be:

As small as is possible while answering questions that caused its construction.
Simple enough that developers can understand it and use it as the basis for a more complete solution.
Provide output information that contains convincing answers about the questions.
Documented as an integral part of the architectural document(s).

You will be required, below, to implement at least two prototypes for Project #5.

Requirements:

Your Software Collaboration Federation (SCF) architectural document:

Shall be written using Microsoft Word or an equivalent office suite that can generate documents in word or PDF format.
Shall use diagrams created with Microsoft Visio or an equivalent tool that can generate diagrams embedded in your document. These diagrams should be delivered in Visio, PDF, or PNG formats.
Each diagram shall be accompanied by text that describes:
- Why this diagram is here.
- What the diagram tells us.
- Conclusions we should draw from its presentation.
Shall, at the beginning provide an Executive Summary that very briefly summarizes the main parts of your architecture, especially its users and uses, parts, and critical issues. The summary does not list the things you did to develop the document.
Shall briefly discuss the goals of the SCF and discuss in more detail the goals of the Test Harness in supporting Continuous Test and Integration.
Shall briefly discuss the structure of the Software Collaboration Federation and then discuss in detail the structure of the Test Harness server, its data, and its events, and how SMS supports Test Harness server in support of CTAI.
Shall briefly discuss the uses and users of the SCF and discuss in more detail the uses and users of the Test Harness. This discussion should consider the impact of uses on the design of the Test Harness server, explore uses beyond those specific to SCF⁷ and decide if any of them need to be addressed in the development of SCF, e.g.: provide for future extensions and additions.
Shall discuss the responsibilities of each of the parts identified above and interactions between them.
Shall discuss issues that are critical for the development of SCF and propose means to address each of the issues deemed to be important for SCF. In doing this you should emphasize issues associated with CTAI.
Shall, at an appropriate place in the document, discuss each of at least two prototypes you've developed. You will develop the prototypes:
- Test Harness Prototype:
  Here, you are using the work you implemented in Project #4, to illustrate how the Test Harness will help SCF support CTAI. For this you will prepare text and illustrations that help the reader understand how the Test Harness server functions and supports the goal of CTAI.
- Critical Issue Prototype(s):
  This prototype is developed to help you answer questions and provide solutions to one or more of your critical issues. What you do and how you do it will be left to you. You will find me reluctant to discuss this in any detail.

Clients and servers are usually identified by machine boundaries but that is not always the case. We may implement more than one server as virtual machines on a single physical machine. Also, we may have more than one physical machine that implements a server. For example, toward the end of a large project testing becomes a very intense large-scale activity and might be carried out on a Test Server Farm.
The Build Server will probably share the same machine as the Repository and so is not shown in Figure 1.
These clients are often processes that run on a developer's machine. SCF clients are not expected to provide compilers and editors and other build tools. Those already exist and would be a waste of resources to re-implement. So a developer's machine would have both the tools and the SCF client software.
Note that Project #5, Fall 2015, focused on use of a NoSQL database that we developed in earlier projects during that semester. This year we will assume that the Storage Management Subsystem (SMS) will be based on the open-source MongoDB. One would expect a system like SCF to have several services around which its functionality is configured, e.g., services for communication, security, storage, display, ...
Each SCF server will have a core set of services that will be common to most of the other SCF servers.
Services like Communication and SMS will contain many parts that interconnect with SCF and that handle it's internal operations, e.g., data storage, query handling, persistance, message handling, and internal error handling.
If we don't look to future expansion of capabilities we may find that after a big expense and effort to build the (sub)system we end up with an inflexible body of software with no growth potential. That's definately not what an architect is paid to create.

What you need to know:

In order to successfully meet these requirements you will need to:

Understand the purpose of an architecture document, which is an elaboration of the Operational Concept Documents you developed in Projects #1 and #3.
Understand the details of the design and implementation of Projects #2 and #4.

Things for which you lose points:

Weak presentation of your discussions, e.g.: inadequate detail and inaccuracies in detail.
Formatting issues: your text and diagrams should convey their information in clear, simple fashion. Formatting should be consistent. If your diagrams and text are inconsistent in format, structure, and tone, it becomes clear that you did a lot of cut and paste from the web and class documents rather than thinking through issues and their presentation yourself.

Things for which you gain points:

I value highly your presentation of original ideas and discussion of original structures.
Your content should provide a broad treatment of a lot of the system parts and behaviors with modest detail and much more detail for those parts you think are particularly interesting and/or important. The broad treatment teaches your reader about the system and its context. The detailed treatment tells the reader about the important things you uncovered or developed.
Feel free to bend the preliminary architecture presented above in ways you think are useful and interesting.