Virtual reality for the observation of oncology models (VROOM): immersive analytics for oncology patient cohorts

Our approach was to be in line with recent studies which showed that egocentric frame of references reduced the mental load needed while exocentric references led to better performance^53,54. We proceeded our development with the combination of both exocentric and egocentric views to utilise the usability of the two designs.

VROOM design specifics

The study’s objective is to match a clinician’s need to understand their patient of interest (POI) by interrogating the POI in the patients’ cohort in the similarity space. The process of interrogating a POI is a comparative exercise in which doctors will want to examine a group of patients similar to the POI with the expectation that this may allow the doctor to expect a common response to a set of treatments. The clinician’s comparison mechanism changes from the patient presentation and blunt pathology test to more profound, more refined molecular information captured in their genes and genetic activity. Our visualisation allows the clinician to identify the patient(s) for comparison (POC). Once POI and POC (s) are identified, the clinician can conduct a detailed analysis of the patient in our visualisation. Currently, the POC(s) selection can either be the nearest ten neighbours in the similarity space or driven by the game theory as an alternative guided analysis. We aim to develop an analytical tool to guide the clinician in their prognosis and treatment regimen development by making these comparisons. Our design specifics will have following characteristics:

• The main visualisation is the 3D similarity space of the patients based on their genomic profile and been clustered based on their genomic similarity relied on the chosen dimensional reduction algorithm as shown in Fig. 1. We will go through the three use cases of the visualisation: (1) Patient details; (2) Patient to patient comparison; (3) Patient to group comparison. The result of the design study has been integrated into the immersive visualisation.

• The visualisation design incorporated tablet-like panels and 2D windows to display box plot, histogram, heatmap, hierarchy clustering dendrogram and tabular form. The ultimate goal of this work is to facilitate the discovery of new knowledge based on genomic similarity and possible application in personalised treatment.

• By working closely with the domain expert, the visual design had undergone an preliminary evaluation with a small number of cancer research experts and clinicians at the Children’s Hospital at Westmead, Sydney, Australia¹⁸. The qualitative comment on the early prototype provided us constructive feedback on our work so that we could improve our current development. For example, the comment on enhancing the ability of highlighting specific and associated individual patients or patient group(s). This comment helped us to improve the visualisation with visual links between the selected patient and the nearest neighbours in the similarity space.

• To reduce oculomotor discomfort in the immersive environment, it is suggested^54,55 that using virtual body and physical movement of body help in reducing such discomfort. In our implementation, we minimise the walking by placing the user interface and data visualisation in front of the user by placing the system in a dome-like environment on a platform.

• We fully utilised immersive multisensory capabilities to engage our users in the visualisation. With the low cost of these VR/AR devices, 3D design has become very common, and with urgent needs of working from home, 3D presence is being developed and tested for daily use. Our user interface mimics the real-world object as suggested by Pangilinan⁵⁶ in the form of a virtual desk. The virtual desk concept has also been applied successfully by Wagner Filho et al.⁵⁷. It creates an environment that the users can intuitively know how to use.

• When crafting our visualisation, the application of these design principles has been integrated and carefully orchestrated. Our design is simple to use and mimic real-world objects to ensure user familiarity. The use of audio feedback is beneficial when interacting in the VR environment to inform the user of the occurrence of the action.

• For immersive visualisation, proper sound design can create a sense of presence, especially in connection with the visual cue^58,59, and it is essential in interaction⁶⁰. Bleeker et al.⁶¹ describe data sonification as the sound rendering technique to provide access to data and in response to interaction events. Massiceti et al.⁶² show improvement in term of task completion time, velocity and other navigational behaviours with the help of sonification. Song and Beilharz⁶³ argue that sonification should be both aesthetic and informative and presented evidence that data sonication improves user experience. However, when designing the sounds, care must be taken to avoid repetition and ensure fatigue avoidance, such as a simple option to allow users to turn both the background sound and action sound on or off⁵⁹. An immersive environment soundscape may have two types of sounds: passive and active sounds^64,65. Passive sound is the background sound or music to create a general atmosphere; while, active sound engages our awareness, usually in response to the event triggered by our action^58,59,64,65.

Based on the sound design principles above, we have implemented three main sonic components in our study: data sonification, user interaction and ambient sound. With the lack of tactile feedback, audio becomes vital in providing interaction feedback to the user. Data Sonification aids the user when traversing a large data set by providing supplementary sonic feedback that is unique to each data point. Similarly, every action taken by the user will trigger a corresponding sound event to affirm the outcome of the action. Ambient sound is used to immerse the user and enhance their sense of presence in the virtual world. Initial sonification prototypes emphasised a high-fidelity correspondence with the underlying data points where possible, mapping sonic frequency, volume and wave shape directly to various data parameters. While it produced an accurate sonification, the large data sets created a highly cacophonous soundscape that was unpleasant to experience and difficult for users to comprehend.

To make the overall sound design pleasant and harmonious for the user, the final sound design for VROOM uses five guiding principles for each sonic component: aesthetically pleasing, congruent with action, congruent with visuals, distinctive signature and spatially informative. The ambient, user interaction and data sonification components all share the same pentatonic scale and musical key, but each separate component has unique instrumentation and resides in a separate frequency and spatial domain. This design allows the user to easily distinguish between any ambient, user interaction and data sonification sounds while maintaining a harmonious overall soundscape. While more pleasant for the user, this approach relegated the data sonification to merely supplement the visualisations rather than providing any new information on its own and this could be any area to readdress in future studies.

VROOM case study: AML

Our visualisation currently is designed to visualise the genomic similarity data for Acute Myeloid Leukaemia (AML) patients with a 2D tablet-like toolset for detailed analysis of the patients. The visualisation adopts the exocentric view for the similarity space for the user to start their inspection of the data; and, if further details analysis is required, the user can either walk into the dataset or use zooming and panning to be inside of the data to take an egocentric view for details inspection. This has been proven to be effective by Risch et al.⁶⁶.

In our Visualisation, the gene expression signature called leukaemia stem cells-17 (LSC17) analysis score⁶⁷ is used for the list of gene-of-interest for AML. This is further clustered and dimensionally reduced into 3D space. The default dataset for the cohort space is “Hovon ThreeRisk” dataset, and the default dimensional reduction algorithm is autoencoder. For our walkthrough analysis, we will use “OHSC ThreeRisk” dataset and autoencoder as the choice of selection. The POI is “aml_ohsu_2018_16-01272”. The POCs are “aml_ohsu_2018_16-01272” and “aml_ohsu_2018_16-00316”. The POCs for group comparison is based on the recommendation from the game theory. For visual identification, our POI is also the currently selected patient, which has a green halo over the sphere, and it is oversized compared to the rest of the sphere, as shown in Fig. 2.

Figure 2 shows the similarity space in the visualisation. The colour of the sphere (red: high risk, yellow: medium risk, and green: low risk) is used to represent the patient’s risk level based on their LSC17 score. LSC17 score indicates the risk of a patient who might not respond well to standard therapy and should be considered for novel alternative treatment⁶⁷. Figure 2 clearly shows that the patient cohort can be roughly divided into three zones labelled, a, b, and c. If a new patient introduced into the system falls into one of the regions, for example, region a, there is a high possibility that the patient has a high risk of rejecting the standard treatment. The details of the patient and the comparison of POI to POC and POCs can be done with the visual analytics tool. The clustering also allows the analysts to choose their POC if they want to look at similar patients or dissimilar patients by comparing patients in the same region or patients from the region a and region c to validate if their treatment regimen affects survivability.

Once the analyst decides the patient to be investigated, in our case, “aml_ohsu_2018_16-01272”, the analyst can drag the patient from the similarity space and collide it with the Patient of Interest Panel for detailed analysis as shown clearly in Fig. 3 with the use of the virtual hand in immersive space controlled by the Oculus Touch controller.

Figure 3 shows the type of visual analysis that can be applied to the POI. The panel is created to have a tablet look-and-feel to tap into the user’s familiarity with the 2D tablet interface. The analyst can grab and move the panel around the immersive environment to organise and maximise their working space. Metadata for the patient is displayed on the left of the panel. In our case, patient “aml_ohsu_2018_16-01272” is a 64-year-old male whose LSC17 score risk is intermediate. The patient’s gene expressions for the LSC17 gene list are displayed in box plot and bar charts on the right of the panel. Figure 3c is a box plot for comparing our POI “aml_ohsu_2018_16-01272” gene expression value to the rest of the cohort in the OHSC dataset. It is noticeable that most of the POI’s gene expressions are within the interquartile range (IQR) of the box plot. Figure 3d shows the POI’s gene expression in the form of a bar chart. An observation that can be made here is that although a gene can be highly expressed in terms of value, for example, CDK6, for our POI, it is not as activated as the rest of the cohort since it is below the Q1 in the cohort. According to GeneCard(R) The Human Gene Database⁶⁸, “CDK6, along with its partner CDK4, are key players in cell cycle progression. The complex has been implicated in several cancer types and is the focus of therapeutic research and development. One targeted therapy for CDK inhibition is palbociclib, which may slow the growth of advanced-stage breast cancers. It has also been shown, in mice, that CDK inhibition may sensitise mutant PIK3CA tumours to PI3K inhibitors”.

This type of in-depth analysis can help the analyst better understand the patient prognosis and potential risk and treatment choices, giving them the ability to drill down to gene-level expression.

Patient-to-patient comparison

Our visualisation allows the analysts to compare the POI to a POC. This would allow the analyst to understand how similar the patient is by using linear regression. Figure 4 shows the process of comparing our POI to two POCs from different clusters in the 3D cohort similarity space. When the analyst’s virtual hand hovers over the sphere, representing patient n, tooltips will provide an overall gene similarity between the selected patient and highlighted patient n (blue). This will give the analyst a sense of the similarity for the patient they are choosing for comparison.

We will continue to use the same POI, “aml_ohsu_2018_16-01272”, for the following 2 case studies. A POC, “aml_ohsu_2018_16-00271”, from a cluster further away from POI, is chosen in our first case study. This is a scenario where the analyst wants to see how dissimilar the POC is from the POI, Figure 4b shows the metadata for our POC, a 22-year-old female with a high-risk factor for LSC17 score. As shown in Figure 4b, the regression line does not fit well due to the dissimilarity of the two patients. The R-square value is 0.179. R-square is the squared multiple correlation or coefficient of determination for our linear regression model. The closer the R-square value to 1, the better the linear regression is describing the data. It is also called the goodness-of-fit measure for our regression model⁶⁹.

For our second case study, the POC, “aml_ohsu_2018_16-00316”, is chosen who is roughly located in the same cluster as the POI, Figure 4c shows the metadata for this POC, a 72-year-old male with an LSC17 risk factor of Intermediate. This POC has an overall similarity with POI at 91.5%, which is higher than the POC in the first case study. The R-square for our linear regression model, at 0.731, also confirm the better degree of goodness-of-fit of the data.

The scatter plot can also provide a visual cue of how many genes are very different for the POI and POC. If the two patients have the same gene expression profile, the linear regression should have a y=x equation.

Patient-to-group comparison

Our key novel innovation is the patient-to-group comparison panel. It is implemented similarly as a tablet. The Patient-to-Group panel is the most feature-rich. The patient-to-group analysis would require the analyst to select a POI and a group of POCs. The POI selection is made using the Oculus Touch button. The system can help select two distinct groups of POCs, or the analyst can pick any POCs of their choice. In the first case, the system can identify the nearest nine neighbours with the highest overall similarity to the POI. Our decision support system engine drives the second type of POC selection. The POCs are visually linked to the POI using white colour lines. The automation of the analysis is used to help identify which combination of three genes out of the LSC17 combo will yield the highest survival rate when analysed separately.

Figure 5 shows the flow on how to use the decision support system engine to assist in patient analysis. The Decision Support System can be triggered by clicking on the “Patient of Interest menu” (Fig. 5a). A panel will be shown with the top 5 combinations of the gene, dimension reduction algorithm and if the combination is a result of Nash equilibrium or social optimal result on the left of the panel. The right panel will have the genes’ description in question extracted from GeneCards®(Fig. 5b). By selecting the different combinations, the highlighted neighbours would be changed accordingly.

This is one of our novel implementations of Overview & details complex analysis which involved looking at the overview of the data while conducting a separate and independent drill-down analysis of the data to allow complex immersive analytics to be conducted. The analysts can then go back to the similarity space to inspect the newly identified POCs suggested by the Decision Support System. To complete the POCs selection for our patient-to-group panel, the analyst can send this data to the panel using the P2G button. For our case study, the selected combination is the top item, which has the gene combination of KIAA0125, SOC2 and AKR1C3, which was found using NMF, and the result was a Nash Equilibrium and social optimal solution for game theory.

The patient-to-group panel will then be activated and be auto-populated with the POI and the POCs, as shown in Fig. 6. The patient-to-group panel will present the metadata for POI and the list of POCs on the right. Figure 6a shows the relationship and linkage between the POI and POCs. The POCs are highlighted with light blue circles. The patient-to-group panel (Fig. 7a) contains 4 visual analytics tool to drive the analysis: HeatMap (Fig. 7b), Patient table (Fig. 7c), Box Plot (Fig. 7d) and a 3D HeatMap (Fig. 7e).

Heatmap is very commonly used in genomic data analysis. It is a fundamental tool in visualising gene expression data to unravel hidden patterns in genomic data⁴⁹. Our visualisation implemented the heatmap with the integration of the dendrogram using a hierarchical clustering algorithm (Fig. 8). Colours are used to indicate the level of expression for the gene (Red: low expression value, Green: high expression value). The level of hierarchical clustering of the heatmap can be changed with the + and − button. Changing the gene level from 14 to 10 will show a different clustering effect on the gene, as shown in Fig. 8. At level 14, the genes were clustered into two separate groups, while at level 10, the gene was clustered into eight distinct groups as shown in Fig. 8.

Box plot is available to analyse the POI and POCs (Fig. 7d). However, the implementation in patient-to-Group is different from those found in the patient-details panel. In this implementation, the box plots visualise the individual POCs and POI gene expression values. This will give the analyst an idea of the overall gene expression for these patients.

The patients’ clinical history is available in tabular format (Fig. 7c). Most of the public dataset does not come with treatment and treatment history, but our OHSC dataset contains the current treatment regimen, which will be very useful for personalised therapy analysis after conducting an end-to-end analysis.

Lastly, a 3D visualisation of the heatmap is available for analysis (Fig. 7e). The visualisation was rendered with the gene expression value to determine the height of the mountain and the valley. It can be accessed with the 3D button. We intend to expand our future work to allow the analyst to walk on the 3D heatmap to experiment with the effectiveness of the future immersive capability of our visualisation.

Walkthrough of end-to-end analysis

In this section, we will do a walkthrough of an end-to-end analysis using our immersive visualisation components such as 3D similarity space, the patient details panel and patient-to-group panel for overview and details analysis to discover the knowledge embedded deep in the data for our POI and POCs (Fig. 9).

We will continue to use the same POI and POCs identified earlier. First, using the patient-details analysis panel, the analyst can investigate the gene-of-interest, for example, CDK6, expression value for the POI, to see if it falls in the IQR range or an outliner (Fig. 9a). The analyst can choose their POCs or allow the decision support system to suggest a POCs group to analyse. In our case study, our analyst wants to understand how to find those nearest patients that are very genetically similar to the POI and have survived the illness. This is what our decision support system was designed to handle.

Once the selection for POCs is made, the analyst can conduct a thorough genetic analysis study using the heatmap with the integration of hierarchical clustering. This will allow them to unravel any hidden patterns in the data (Fig. 9b). The heatmap shown in B was based on the POCs suggested by the decision support system, which was based on the “KIAA0125”, “SOC2” and “AKR1C3” gene combination, as it is clearly shown in the heatmap that for those items (indicated by the white arrows), the colours are very similar across all the patients. The analyst can then concentrate their gene expression differential analysis on the rest of the genes.

After a thorough heatmap analysis, the analyst can check the medical history of all the patients (Fig. 9c). Taking a quick look at the table, the analyst will find that only two patients from the cohort passed away, our POI and patient “aml_ohsu_2018_15-00371”. The last regimen given to our POI was not the same as the rest of the POCs. This might give our analyst an idea of what was prescribed to POI and how to form a hypothesis to be tested biologically. It is also noted that patient “aml_ohsu_2018_15-00371” was given the same regimen, “HiDAC”, which is similar to the other three patients, but the patient did not survive. However, patient “aml_ohsu_2018_15-00371” is significantly older than those three patients. The former was 74-year-old, while the other three falls under the age of 60s. According to Cancer Institute NSW⁷⁰, the treatment efficacy “for patients aged 60 years and younger the probability of survival after four years was 35% in the 100 mg/m² group (CI 95% 27–43%), 40% for the 400 mg/m² group (CI 95% 32–49%) and 52% in the 3 g/m² group (CI 95% 44–60%) (p = 0.02)” and the side-effect for “high dose cytarabine chemotherapy was much more poorly tolerated compared to the lower doses in the above 60 years of age group”. There might be a possible linkage that is worth further investigation by the analyst. The analyst can also go back to the heatmap to visually inspect if there are significant differences in the four patients treated with “HiDAC” to look for a pattern.

Lastly, the strength of our immersive visualisation lies in the fact that there is unlimited working space to allow the analyst to conduct in-depth and complex analysis without leaving the system. As shown clearly in Fig. 10, the analyst can stack the panel vertically on top of each other to have a clear visual inspection across the panels and, in comparison, to the 3D similarity space.

User qualitative assessment

We have sought user assessment of our prototype with twenty participants (10 females and 10 males) representing clinicians (20%), genetic experts (40%), bioinformatic experts (20%), and health-related research students (20%). The greater number of potential participants were identified and invited by emails across two universities and a hospital in Sydney, Australia. Most of the participants (75%) had no or little experience with any VR device prior to the study. Each participant was instructed to complete a set of tasks based on a clinical scenario using our VROOM tool to assess the ease of usability, accessibility and interpretability of our prototype. For examples, the study asks the participants to interact and manipulate the visualisation to create comfortable views, select patients of interest to provide necessary visual analysis. The analysis includes examining the patients’ details as well as carry out comparisons for patient-to-patient and patient-to-group via graphical charts. A post-session questionnaire and verbal debriefing were carried to collect the participants’ feedback on the usability and the acceptability of the visualisation in the VR environment.

Results from our assessment showed that the participants highly regard the potential usefulness of the VROOM tool for distinguishing patient to patient comparisons based on the modelling of their genetic and biomedical information (10% very useful, 55% useful and 25% moderately useful). Specifically, 75% of the clinicians indicated that VROOM would be very useful for the patient analysis after their experiment with the clinical scenario. Qualitative comments showed that the participants expressed excitement that our VROOM VR environment allowed them to walk within the data, press a button, and select a patient for detailed assessment.

Regarding to the visualisation in the VR environment, we received very positive feedback from the participants on the accessibility of the current design in navigating the environment. Most of the responses indicated that the visualisation had an impact on the user perception and user experience (moderate impact 35%, impact 30%, and extremely impact 15%). We received constructive and positive qualitative feedback from the participants on the graphical design of the avatars where they helped them identify quickly additional information about the patients, such as their gender. Critical comments were also received on the design, such as the use of the red-green colour scheme not being so user friendly for colour deficient users and the need to showing additional information on the avatars, such as Body Mass Index (BMI), risk, and age. These limitations will be addressed in the next improvement of our VROOM prototype.

Most of the participants indicated that they noticed the sound during the trials which as instructive to them when completing their tasks (moderate helpful 15%, helpful 45%, and very helpful 20%). Qualitative feedback from the participants demonstrated that a “ding” sound which was congruent with action were helpful to bring to their attention that they have done a task correctly. The background music added positive experience in VR and helped them sustain engagement in the task.

We also identified that participants navigation in the VR space allowed them to investigate and find the information they need. 45% of the participants indicate that VROOM is “easy” to “very easy” to navigate and find the necessary information without any support, 25% of the participants could easily navigate with some help from the instructor, and 30% of the participants found it initially quite hard to navigate. This outcome reflects that our navigational design is in the right direction to meet the goal of the prototype, but there is a need for further improvement to a more intuitive navigation. In addition, sufficient training and practice are crucial to users who are particularly not familiar with VR and its navigational controller.