A favorite feature that Xray provides is not only the ability to pass or fail individual steps but also the ability to comment and add defects and evidence to those steps.
Tiffanie Neal, COLSA Software Analyst II, Quality
COLSA Corporation is an innovative technology integrator providing services and solutions to the DoD, Intelligence, and Federal Civilian agencies. We specialize in serving our customer markets in the areas of Software Engineering, Information Technology, Information Management, Systems Integration, High Performance Computing (HPC), Data Center Operations, and Test & Evaluation. As a business unit within COLSA, the COLSA Software FactorySM (CSFSM) is using the Xray tool for testing all products. The personnel in the COLSA Quality Department that evaluate the CSFSM adopted the Xray tool to have the ability to integrate their audits within the CSFSM development environment.
The CSFSM is an agile-based software development team that utilizes Jira, Confluence, and BitBucket to conduct their projects. They adhere to the CSFSM Factory Agile Software TeamSM (FASTSM) policies and procedures to create products for their customers. The COLSA Quality Department (Quality) is tasked to evaluate the CSFSM for compliance to the FASTSM policies and procedures and CMMI DEV Level 3. The Quality auditor uses documents and spreadsheets to track and perform these audits. An “Opening” meeting will be held with the stakeholders involved in the audit process. Results and final reports from the audits are stored within databases inaccessible to the CSFSM. Final results are then communicated in a debrief or “Closing” meeting with applicable personnel and/or through email.
The interaction between the two departments resembles a silo mentality (Figure 1: The Silo Mentality – Software Development Team vs. Quality Department). Both departments have separate environments and utilize their own workflows, which result in the following issues:
The silo mentality between the two departments is not ideal considering their collaboration is imperative in the interest of the end product and the ability to meet the requirements of the FASTSM policies and procedures. It is apparent that the two departments need to be in the same environment and adhering to a shared workflow. This is accomplished through Quality’s adoption of the Xray tool to perform audits.
Figure 1: The Silo Mentality – Software Development Team vs. Quality Department
The idea for the Quality Department to adopt Xray derived from the CSFSM using the tool for their functional testing of products. It is apparent that Xray is a test management tool, not an auditing tool; however, there are notable similarities between a functional test and a quality audit.
Figure 2: Similarities of a Functional Test and Quality Audit
In Figure 2: Similarities of a Functional Test and Quality Audit, both tasks are the responsibility of an external party that did not develop the product or perform the process. Those individuals verify requirements of the customer are met and procedures are followed. Both functional tests and audits are organized in a similar format, which is a major contributing factor on the selection of Xray. Failed tests result in bugs that the project team must correct before the end of product’s lifecycle. Failed audits result in audit findings that create action items that the project team must resolve before the end of the product’s lifecycle. Both bugs and action items contribute to the growth of technical debt.
Figure 3: Audit Structure in Xray
Through the usage of Jira and Xray, the Quality auditor is able to create audit templates using the Jira issue types “test set” and “test” (Figure 3: Audit Structure in Xray). All audit templates are stored in Jira in a project space for “Quality” that has limited access to prevent unnecessary changes. Audits are easily performed through the creation of a “test execution” Jira issue type that ingests the “test set” or audit template from the Quality Space (Figure 4: Performing Audits in Xray). The “test execution” is created and performed in the applicable Project space so project personnel can access the results in real-time. Audit findings found during the evaluation of projects are also filed to the applicable Project space in Jira.
Figure 4: Performing Audits in Xray
Through the utilization of Xray, Quality is able to relocate into the CSFSM environment for all audits that are applicable to software development projects. The features of the Xray tool allow the Quality auditor to perform its tasks seamlessly with little adjustments. Additionally, Xray provides a number of benefits compared to the previous environment described in the Challenge section, see Figure 5: Benefits of Using Xray to Perform Software Development Team Audits below.
Figure 5: Benefits of Using Xray to Perform Software Development Team Audits
A favorite feature that Xray provides is not only the ability to pass or fail individual steps but also the ability to comment and add defects and evidence to those steps. This feature allows project personnel to review the requirement that failed, why it failed (through the comments), and the associated action item that is linked to the step. The traceability is key for reviewing audit results with team personnel and other applicable stakeholders.
When the Quality Department adopted Xray as its tool to audit the CSFSM for compliance, the interaction between the two departments moved from the Silo Mentality to a unified workflow (Figure 6: Innovative and Integrated Workflow). The CSFSM and Quality auditor assigned to perform audits are able to collaborate in the same environment using a new innovative workflow.
Figure 6: Innovative and Integrated Workflow
This new workflow has provided solutions to the previously stated issues (Figure 7: Results of an Integrated Environment). The results of moving to an integrated environment are that information is accessible to both the Quality Auditor and Project Team, which contributes to continuous communication and increased collaboration. The Quality auditor and project teams are able to work together to prioritize action items from audit findings that contribute to technical debt. This collaboration results in action items being prioritized and allowing the project team to accomplish technical debt early instead of being bombarded randomly throughout the projects’ lifecycle. The Quality auditor becomes heavily involved in the projects and is seen as a benefit to the project teams instead of a burden.
Figure 7: Results of an Integrated Environment
As mentioned in Figure 5: Benefits of Using Xray to Perform Software Development Team Audits, the adoption of Xray provides advantages to Quality as well. As the Quality Department continues to utilize this tool to perform audits, critical historical data is being generated and retained for analysis. Through the increased amount of audits/test executions being performed, the Quality auditor is able to analyze results and determine improvements that need to be considered either at a team or process level. Xray provides the capability for Quality to create and track metrics and trends for analysis to improve the CSFSM (Figure 8: Metrics and Trends).
Figure 8: Metrics and Trends
The utilization of Xray has given the Quality department the ability to improve their communication with the CSFSM. The previous process would require the Quality auditor to perform an audit and then schedule a meeting to discuss audit findings. The CSFSM would then have 30 days to correct the audit findings and present to the Quality auditor for approval. The new process with Xray allows the Quality auditor to perform audits and submit findings as action items concurrently. This ability has reduced the average time for the CSFSM to receive the findings by 96.08%.
This capability allows the project to review and close out action items in real-time. If there are questions or comments regarding the action items, those can be logged in Jira for the Quality auditor to respond to within a recurring two week interval and there is a standing review time of action items every 2 weeks per the CSFSM process. The average resolution rate of audit findings has decreased by 7.92% due to the Quality auditor and the CSFSM receiving updates instantly by working in the same environment.
The implementation of the Xray tool within Jira has given the Quality auditor the ability to perform audits on multiple areas of the projects and processes very thoroughly while minimizing the amount of intrusiveness on the project team. The new environment allows the Quality auditor access to the project team’s processes and information while consuming less time of the project team. Additionally, the individual project team’s awareness increases from audit findings being filed in their environment for their review.
Xray has given the Quality auditor the ability to embed themselves within the CSFSM project teams and processes, while continuing to meet and exceed its requirements of ensuring compliance to customer’s requirements and the FASTSM policies and procedures. A new workflow and process has been developed through the usage of Xray and Jira, which in the future can be expanded to other departments within COLSA.
