About Caris Life Sciences®
Caris Life Sciences® (Caris) is a leader in precision medicine. Their molecular profiling tests help Oncologists diagnose and treat their cancer patients. Caris is also deep in clinical research, supporting the development of tomorrow's treatments. With more than 1,300 employees, Caris has offices in the United States, Japan and Switzerland and provides services throughout the U.S., Europe, Asia and other international markets.
Caris has a lot of software development activity as they deliver Caris’ laboratory management systems and other operational and clinical support systems. They have more than 50 developers, around 20 Software Quality Assurance staff, and often use external contractors to expand their development capacity.
The challenge
As Caris gained more recognition and was moving to mainstream clinical practices, the company found itself in new territory. The demand for their tests had skyrocketed and exceeded their lab capacity. Their systems were slowing down their scale-up. They needed a laboratory management system that was a league up from what they had. The new system needed to be much more automated and meet FDA medical device regulations.
As they planned the ambitious project of developing their new system, the team at Caris assessed their strengths and gaps. They had a very strong software development team. Jira has always been at the heart of their development activities and was carrying them through their Agile sprints. With this setup, it was easy to add external contractors and increase capacity.
But these practices did not deliver on the FDA regulatory expectations. To approve their lab management system, the FDA expected massive supporting documentation, including:
- Detailed requirements specifications;
- Documented tests that prove complete coverage of requirements.
The time to market is so critical for cancer patients and for Caris. The challenge for Caris was how to meet the regulatory challenge without slowing down their development teams.
The solution
“It was easy to acknowledge that the team needed a better tool...We decided to roll Xray out in an experimental project. I built the issue screen schemes and the issue types and the workflows. We also created some knowledge base articles on Confluence. This eased people into using Xray.”
Tony Sexton - QA Manager at Caris Life Sciences
To solve this challenge, Caris brought on board new tools and new experts like Lee Pfortmiller, a senior Business Analyst (BA). His goal was to set up a good process for requirements management. The second expert is Tony Sexton, the manager for Software Quality Assurance. He needed to bring their testing practices to be compatible with FDA imperatives.
Lee, and Tony, with the rest of the Caris team, built a compliance envelope around their software team. The business analysts team took ownership of capturing the requirements specifications and feeding the development backlog.
At the other end of the process, Tony and his team ensured every implemented Story had test coverage. When implementation and testing are done, it loops back to the business analysts. They verify that the traceability from requirements remains complete.
The idea was that the compliance envelope will be in sync with the continuous flow of Agile Sprints. If this is achieved, the impact on the development process and the skill-up required from developers was minimal. But, for the compliance activities to keep up with the development speed, they'd need the right tools. Tools that could deliver a speedy BA and Software QA operation.
“I used Xray in a prior role, and I knew how to install and administer it (...). When I came to Caris, some of the QAs I managed didn't have a test management tool. We had around 20 Software QA professionals, and (...) some of the work was manual, and (...) that left too many opportunities for human error and was very labor-intensive.”
Tony Sexton - QA Manager at Caris Life Sciences
Unlike Tony, Lee did not have a tool that he had used before to fit the requirements' management task. As they used Jira and Confluence, he wanted to find something that would integrate with this toolset.
“We all love Atlassian tools. We've been using those for decades. At the time, we were transitioning from an on-premise site to the Atlassian Cloud, reaffirming our commitment to that platform.”
Lee Pfortmiller - Senior Business Analyst at Caris Life Sciences
Lee found a webinar where RadBee presented how to manage traceability in Life Sciences. That’s how he discovered Jira Snapshots for Confluence.
“(...) that’s exactly what I was looking for. Then setting this up was easy. Working with the Regulatory & Quality Assurance team to really nail down this solution that Jira Snapshots is very much a part of because of its ability to pull data from Xray. The solution that we have right now with more powerful tools, including structure, is incredibly useful.”
Lee Pfortmiller - Senior Business Analyst at Caris Life Sciences
Compliant development process with Xray, Structure for Jira and Jira Snapshots
Compliance for the development process in Caris, relies on these three apps:
- Xray Test Management for Jira: to enable an efficient Software Quality Assurance flow;
- Structure for Jira: for requirements management. Especially for the ongoing monitoring of the traceability between requirements and stories;
- Jira Snapshots for Jira: To have no-burden reporting. The coverage and traceability reports are matching the expectations and habits of FDA inspectors.
The team uses Structure for most of the requirements’ management, the issue with this app for them is that it lacks complete coverage and traceability reports out of Jira. This was the reason they chose Jira Snapshots in the first place, and the possibility of Xray integration provided them with a complete development toolkit that meets compliance requirements and regulations.
“With Jira Snapshots, we can query level one and then pull in all associated issues in level two and so on with more levels. And the integration with Xray is incredible. There's no other tool out there that does precisely that. We have made that discovery of how powerful this is for companies like ours when you bring these tools together.”
Lee Pfortmiller - Senior Business Analyst at Caris Life Sciences
Their compliance and development process now flowing in sync. True to the original intention, the development team dictates the pace. A typical scrum team will include a project manager, business analyst, software developers, and Software Quality Assurance staff.
Backlog grooming
The business analyst starts requirements at the level of the “User Requirements.” Capturing the gist of the requirement from the users perspective. In Jira, this is a “Requirement issue,” which is a custom issue type. From a traceability perspective, these trace down to the Stories.
Business analysts provide the detailed design specifications in the user story. The user stories are formatted pretty much the way they always were. They follow a consistent format (for user stories) with a value statement. They also have a lot of extra information like the dependencies prerequisites. This is information that the developers and Software QA need. The story also has the acceptance criteria that they write in Gherkin. Having the Gherkin code makes the test cases easier to build in Xray.
Because the process is still young, and the team is expanding rapidly and not all are familiar with the process, sometimes stories are created through other channels. Lee uses Structure for Jira to spot such cases as stories that pop up and are not associated with requirements specs.
“Structure is helping us to maintain that. I can quickly come in very easily and associate them to a Requirement (...) so that I don't have any orphan stories sitting out there that might get us into trouble if they weren't part of our traceability. ”
Lee Pfortmiller - Senior Business Analyst at Caris Life Sciences
Benefits of the integration during the Sprint Cycle
“We're having refinement meetings. SQA understands what we're building, and they can immediately begin creating their tests in Xray while the developers are getting ready to start developing. It's a very efficient process where QA is not having to wait for the developer before they start creating test cases. Everything's happening in parallel at the same time.”
Lee Pfortmiller - Senior Business Analyst at Caris Life Sciences
The Scrum team has SQA (Software Quality Assurance) members and Xray gives the team a unified place to build and maintain their tests, and also to keep track of their testing activities. Some of the benefits the team noticed after using Xray were the following:
Improved collaboration between teams
They feel like it helps them collaborate better amongst themselves. When we have multiple people on the same project, it helps their collaboration so they are more effective as a team.
Xray integration with Jira aids in product development
Xray integrates with Jira so that it's the same platform for development and for testing. Granted, the way they built their scrum boards, they don't include some of the statuses on the test items like test cases and preconditions because they don't want those to show up on the board. Those are easily accessible on Xray’s test repository board.
Better support for coverage and traceability
The SQAs are able to create and manage their test execution directly on the board. With Xray and the new processes that support our coverage and traceability reporting, the teams are pretty good about analyzing reports themselves.
Usage of Xray reports to solve bugs
The SQAs use a lot of Xray tools to manage their strategy. They run a coverage report targeted to a specific Sprint or a fixed version or even targeted to a JQL filter, and they can see the test cases and test executions gaps. The two reports used more often are the coverage and traceability reports that ship with Xray.
Post Sprint (Closing the loop with requirements management)
With Jira Snapshots, they have a Confluence page with each of the reports they need. The layout is the same as they need it to be to submit to the FDA. With one click, the team can update the traceability report and ensure that all Jira items are linked and that the traceability is complete. As the software development is progressing from Sprint to Sprint, there's no compliance debt that is building up.
The results
- A compliant process that does not slow the development team. That was their ultimate goal. They needed a process that is FDA compliant, but at the lowest cost on the Agile development processes;
- Easy to make reports. The FDA submission requires specification reports and traceability reports. Jira Snapshots compiles these reports from the Jira and Xray data, avoiding burdening the team;
- Shoot up in test capacity, without increasing the team. With Xray the Software Quality Assurance team's ability to collaborate with the dev teams improved, reducing the intensity of labor and making it easier to provide test coverage;
- Reduced time to generate, store, maintain and report on testing activities. The team is equipped to do a better job and they are satisfied with the tool;
- Automated tests are connected with the compliant framework. One of their future plans is to connect their growing collection of automated tests with Xray.
“We added a couple of custom fields to the Test issue in Jira so that the QA can specify whether we need to automate this Test Case. If the Test is a candidate for automation, there is an extra field that keeps track of the status of the development of the automated Test. This helps manage the test automation effort.”
Tony Sexton - QA Manager at Caris Life Sciences
The team at Caris Life Sciences plans on integrating the automated repositories with Jira and Xray in the future, so that when automatic tests are executed, the results flow into Jira and Xray and become part of the coverage and traceability reports.
The Xray and Jira integration benefits won’t stop at this stage of their process, and since it allows them to meet compliance requirements and regulations, the team feels a continuous adjustment to the process will be best.
Xray and Jira Snapshots helped Caris Life Sciences reach their compliance goals
Thanks to Xray, QA teams and developers improved their collaboration, and have access to tools that allow them to deliver solutions quicker and more efficiently. They have access to full reports with traceability, so they can easily communicate between teams the main gaps found along the development cycle.
FDA submissions that required specifications reports and traceability receive full coverage, which allows them to meet the compliance requisites in their development cycle.
With Xray’s ability to provide quicker delivery and essential reports, Caris Life Sciences scaled in test capacity without having to grow their Quality Assurance Team.
