Links to final source code (github repo) @Mashira Farid

https://github.com/samztz/SENG3011_GeeksForHDs

Open

TEAM

@Tingzhuang Zhou @Avijit Prasad@Mashira Farid @Zifan Wei @Lin Thit Myat Hsu

APIs

Note :- To see the interaction with the frontend look at Software Architecture Figure 2

• https://covidactnow.org/data-api

  • 

  • Risk metric is received which is displayed on the risks maps. Risk metric is determined by population of the county, weekly infection rate, staffed hospital beds and ICU beds in the county and scaled in a range from 1 to 5.

  • Vaccination ratio is received by counties which is later converted into vaccination percentage and displayed with counties.

  • Weekly covid cases by counties are received which is used to determine community threats in each county.

• https://github.com/covidcaremap/covid19-healthsystemcapacity/tree/master/data/published

  • Following dataset is used to receive hospital beds data which is later added to risk data. All the calculations and preprocessing done in. https://github.com/samztz/SENG3011_GeeksForHDs/blob/main/PHASE_2/epi_frontend/data/data_preprocessing.ipynb which produces final data in this format:

• https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties-recent.csv

  • New-York times API used to populate charts in every county

Data

Data is available at Data folder in github.

To get final preprocessed data https://github.com/samztz/SENG3011_GeeksForHDs/blob/main/PHASE_2/epi_frontend/data/hospitalandrisk.json

Feedback

Deliverable 1

Through feedback we realised our understanding of article and report data that we were supposed to produce is not correct. We were also asked to document our project management better through Grantt charts and more detailed communications which we included in the final report

Deliverable 2

We were advised to test the APIs in a correct manner. We were told about the structure or articles and reports we were supposed to get which helped us in our implementation.

Deliverable 3

In the first Demo we realised our use case and requirements were not properly defined. We focussed on a broad range of users and our website had too many features that didn't cater to a specific target audience.  Our main goal from the start was to create a website that caters to policy makers. To help our target user do resource management and surge planning with the aid of our website. We initially decided to make our websites for analysts as we thought we would provide them with different visual aids (charts and map) with some raw data to help them do their analysis better in outbreaks.

After the demo, we got feedback from our mentors that the idea to provide visual aids and data is good but the target audience does not require these services as they specialise on them. We took our time and through the feedback changed the whole website with a more clear target user. Adopting the age old 'Do not fall in love with your code' approach we started our project from ground up but this time with a more clear use case and service we were providing.

We removed redundant features and shifted our focus to policy makers in countries, specially developed countries, so they can get relevant data and visualisations to help them. We also got an idea to focus on small suburbs rather than countries to better represent our data from our mentor. Through this we focussed on making one map but made it as detailed as possible with enough insights so a policy maker can make most of his resource decisions through it.

We also preprocessed datas from different sources to give useful insights to the user. In the end, the feedback process helped us to understand the importance of having a Use Case in mind before implementing features.

Application Architecture and Tech stack

By the time goes by, the software architecture is constantly changing over time. The change base on the iterating requirements, changing end-users, and also the UI design.

Initially, we wanted to achieve features like user authentication, bookmark of queries, subscription, warnings, data visualizations in list/map/charts, case predictions. And we are targeting the end user of data analyst. This initial software architecture be like:

We have our node.js backend connected MongoDB atlas for features like user authentication, bookmarks feature, and we have the NextJS/React/Redux/MUI front end to build the client side applications.

But soon we realize it is very overwhelming for us to complete everything in demo, and we didn’t do well in week 8’s demo, therefore after we deeply reconsider the feedback from the demo, we simplified our application architecture from multipage application to simple page one and focusing on the user interaction of data that will benefits to customer. the reworked architecture is showing blow:

The architecture is more straight forward that we have our NextJS rendering the React simple Map, and the side information Card, the card contents data plus Recharts case chart, each time user click into different counties in the map, the Card will be filtering data from API calls by useState hooks.

Backend - NodeJS, ExpressJS

• NodeJS: an open-source, cross-platform runtime environment for JavaScript built on Google’s V8 engine

  • We chose to use NodeJS as it is very simple to use and easy to learn.

  • Since Node is event-driven, it can also enable asynchronous programming, which provides great performance compared to Python

  • Since NodeJS is open-source and cross-platform, everyone in the team could develop the app regardless of their development environment

  • By using NodeJS, we can also use NPM (node package manager), which allows us to install numerous libraries to use in our code to help in development. ExpressJS is an example of one of these libraries which is installed using NPM

• ExpressJS: a backend web application framework that's used to design and build web apps quickly and easily

  • We decided to use Express as it allows us to create RESTful routes easily. This is particularly useful as we will be developing our own API as part of the project, which we will need to create custom routes for

  • Since Express is an very popular NPM library, it works well with Node

  • Express also supports JSON, which we plan on using for our API to return results

Frontend - ReactJS

• ReactJS: an open-source frontend JavaScript library

  • We chose ReactJS for the frontend because it supports the use of components, which allows us to reuse existing code instead of having to rewrite similar features, which in turn allows for faster development

  • React is also better in terms of performance compared to other frameworks like Angular, as it only updates components in the DOM whose state has changed, instead of updating every component again, making our app lightweight and flexible

  • React also has plenty of documentation and online resources, so the members of our team who were unfamiliar with it would be able to pick it up quickly

Development and Deployment Environment:

Development - Cross-Platform

• Since everyone in our team is using different operating systems on our personal machines, we ensured that all the technologies we chose as part of our tech stack worked across different platforms

• This allows everyone in our team to work on our app in the development environment they were most comfortable with and had access to, which in turn leads to more efficient development of our app, compared to setting a specific development environment that not everyone may have access to

Deployment - Heroku (later switch to AWS if time permits)

• Heroku: a cloud platform as a service for app deployment

  • We chose to deploy our API using Heroku as it only requires a few steps in order for deployment. This will make it quicker and easier to deploy our API compared to other platforms such as AWS

  • Additionally, since our API is small, it doesn’t need all the functions that AWS provides, making it much simpler for us to deploy it using Heroku

  • However, if time permits, we plan on deploying our API using AWS

Database - MongoDB

• MongoDB: a cross-platform, NoSQL database

  • Initial database options are: MongoDB, PostgreSQL, SQLite

  • We chose MongoDB as our database as our app isn’t very big, so we didn't need to use SQL

  • MongoDB is more scable as it will initial a new data fields whenever we create the documents in database, hence suitable for agile iteration when adding new feature later.

  • Additionally, MongoDB integrates well with NodeJS, and stores data in JSON format, which matches the format our API will output in

Testing - Postman

• Postman: API testing application

  • We chose Postman over other tools such as Insomnia to test our API as a few of our members had experience using it

  • Postman also provides the ability to collaborate as a team to write, store and run test cases

Specific Libraries:

Library - PuppeteerJS, Morgan(logger), Winston(Logger), Nodemon, CORS (TODO)

• PuppeteerJS: an open-source NodeJS library

  • We use PuppeteerJS for scraping the website as the library allows the backend to run a headless (a code-only) version of Chrome to load the websites on.

  • A headless version of Chrome doesn’t use as much resources and as such can be easily run in any server, such as the hosting servers that we are using.

  • PuppeteerJS also have high-level API control of the Headless Chrome browser via the DevTools Protocol. This allows for the code to read the structure code of the website directly and to access the required parts of the website.

• Morgan: HTTP request logger middleware for Node.js

• Winston: Another logger for both develop and production environment.

• CORS: NPM package for solving Cross Origin Resource Sharing problems

• dotEnv: abstracting environment variable and url configurations in .env file, making swaping of environment between development/production feasible.

UI design - Material UI

• Material UI (MUI): an open-source frontend framework for React components based on Google’s Material Design

  • We decided to use MUI as it would make the development of our app quicker and easier. Since MUI defines its own components, we can use them to quickly design the UI of our app, instead of spending time writing our own CSS classes

  • Since we had decided to use React for our frontend, and MUI uses React components, it was only logical we used MUI over other non-React frameworks such as Bootstrap

  • Additionally, some of our team members already had some experience using it. The extensive documentation also means it will be quick and easy for the team members who haven’t used it before to pick up, as well as debug

  • By using MUI, we can keep our frontend UI theme consistent, and even customize it to suit our needs

Recharts JS - A composable charting library built on React components

• We choose Richards over the other visualization kit, because the document is easy to read so we can easily implement the chart in our own

• While testing the charts framework, we found out Recharts includes built in neat animation by default, the animation will further enhance user experience

React Simple Maps

• Create beautiful SVG maps in react with d3-geo and topojson using a declarative api.

• We choose the React Simple Maps as our core package, over the other maps like google maps, because it is a React map library that allow us to customize a zoomable, Composable map component, that help use develop the functionalities like click to switch county display.

Key Benefits

Our first main key benefit was the total overhaul of the design compared to our old week 7-8 design. While that design had some benefits, with help from our tutor we managed to refine our ideas and move from an application which did a lot of multiple things decently to a much better application that did one thing very well - all done within only 2 weeks.

Another benefit we found is the singe-page design of the website. Compared to other designs we’ve seen, a single-page design like the one we have reduced the load time of loading multiple websites and making the user experience much faster and smoother for the user to use.

The website’s map’s ability to highlight certain information also makes it easier for users to gather and display information in a quick and efficient manner compared to other websites, and since it’s all accessible on one page, switching between different maps is quick and easy. The fact that each of the areas on the map can be interacted to display as more information, as well as the graphs, makes our website a potent source of information at any given time.

Use cases:

We must flatten the curve to give our healthcare system more time to prepare for this surge of patients. Preparation means enough space, staff, supplies, and systems at the right places and times to meet patient needs. Through mapping and geospatial analytics, we provide situational awareness and foresight for health systems, policymakers, emergency managers, suppliers, and the public to plan and coordinate preparations between facilities, counties, states, or nationwide.

Use case 1:

A policy maker wants to know the risk level of his county in the surge of the pandemic to recognise any resource needs he needs to address

Requirements:

1. Visual mapping of risk on every county of USA

2. Displaying risk on every county

Requirements Fulfilled:

1. Risk data mapped to a map

2. When a policy maker checks the severity of one specific county, there is one customised colour bar with discrete number as indicator shown below the risk level graph. It can use colour to differentiate the severity among all counties

3. Policy maker can hover on map to display risk level

Limitations:

Problem:

Data not called dynamically from an API

Solution:

Host the python file in lambda function which in futures can be called to get processed data.

Use case 2:

Policy maker can also look into his county for more information which will give him insights of the risk being shown. For example he can look the vaccination rate of the county and increase vaccination programmes to solve the issue is the rate is less.

Requirements:

1. Letting the user click on the chart to zoom in the county which displays more metrics of the county

Requirements Fulfilled:

When a policy maker clicks one specific county on the graph, there is one side bar on the right-hand side pops up, it contains three parts, first part above is including county name and risk level latest update date. Second part middle includes a status bar that shows the severity of that county based on the risk level. As well as infection rate and other related information. Third part at the bottom exists one line graph which records the change of the number of epidemic cases in past few days. This detailed information can bring a clear overview of the county during pandemic which gives policy maker makes appropriate decision.

Use case 3:-

If a policy maker sees his county is in risk , he may want to get an idea of hospitals beds in his county to accommodate more patients in the coming future. That includes total beds, staffed beds and ICU beds to recognise where he should focus to manage his resources to minimise risk. These metrics are given with a ratio to population in the county. For example a county can have a high risk and sufficient beds, but it has less ICU beds which is seen through the charts. Also these charts can give insights about other counties so counties with high resources and less risk can be recognised by the user and the policy maker can communicate with them for resources

Requirements:-

1. User can change the map types to see different hospital beds ratio in different counties.

2. Hospital beds ratio mapped to counties with an appropriate colour scale

Requirements fulfilled:-

When a policy maker checks how many empty beds left, the map type would be switched to Staffed Beds graph. There is one customised colour bar from light colour to dark colour to indicate how many beds left in each county. It could make it possible for distributing the resource between counties because the complete data analysis of staffed beds and risk level provided in the visualisation. The policy makers can compare the risk level of the county they live in with their adjacent counties, as well as comparing the staffed beds left between them. Thus, it could show the possibility to borrow some beds from one county to another which dealing with the insufficient supply of beds for that county.

Limitations:-

Problem:

Data is processed from API’s but is not dynamically available

Solution:

Create lambda function of python file and receive data from hosting an API from it

Use case 4:-

If the policy maker has to do his own surge planning through tools available on different websites like

https://www.euro.who.int/en/health-topics/Health-systems/pages/strengthening-the-health-system-response-to-covid-19/surge-planning-tools/adaptt-surge-planning-support-tool they are able to download data in our website

Requirement:-

Provide link to download data to the user

Requirements fulfilled:-

When a policy maker first go to the page of our website, it would pop up a model which contains the da link and confluence link. Thus, they can download the data as they want.

Other Limitations for future updates:

Problem:

Users cannot be alerted with risk level changes. 

Solution in future updates:

When risk level changes which can be achieved through a setInterval method in use-state module of React. Users get notified through email with EmailJS.

Future use case:

Our main goal of the project is to increase the maps to developing countries which needs tools like these the most. As we know in the previous pandemic countries like India had a problem with hospital beds and medical supplies. Also these tools focus on small areas as people managing those areas do not always have a data analyst to aid with these important projections. Visual maps makes those policy makers life easier to recognise in detail if they need to act fast with their resource management.

Responsibilities of Each Member

Unassigned Responsibilities (done by the whole team)

• Project Setup

• Github Repo Management

• Presentation drafts

• Research to refine Use cases

• Finalising features of the system

Jira roadmap

Deliverable 1

• @Lin Thit Myat Hsu mapped out the report, and was tasked with keeping everyone up to date and coordination of resources, and as such mapped out the project plan.

• @Mashira Farid and @Avijit Prasad were making plans for the API and the basic code, while @Tingzhuang Zhou was tasked with justifying the language and development environments.

Deliverable 2

• @Mashira Farid made the scrapper

• @Lin Thit Myat Hsu made the tests

• @Tingzhuang Zhou made the APis and devided the tasks

• @Avijit Prasad helped with the scrapper

• @Zifan Wei helped with the scrapper

Deliverable 3

• @Avijit Prasad decided use cases and devised work, he also worked on services page that adhered to the use case

• @Mashira Farid made the UI/UX design and made Home page, Reports page for the demo

• @Tingzhuang Zhou did project setup and backend

• @Lin Thit Myat Hsu Did risk analysis and worked on landing page

• @Zifan Wei Refined use cases

Deliverable 4

• @Avijit Prasad finalised use case and allotted everyone tasks

• @Tingzhuang Zhou made the side bar

• @Mashira Farid made the maps and UI/UX decisions

• @Lin Thit Myat Hsu Made popup page and risk analysis

• @Zifan Wei Refined the final use cases and did requirement analysis

Project Summary

Major achievements

Choropleth map

One of the main features of our final website is the interactive choropleth map of each county in each state in the US. It displays different types of data, such as covid risk level, the ratio of the population vaccinated, and the number of staffed and licensed beds, including ICU beds. The user can select which data they wish to view via a dropdown menu, and updates the map and colour scale legend accordingly. Users can zoom in and out, hover over a county to get its name and the value of the data being shown, as well as click on the county to get more information. By having this map, policymakers can coordinate with other counties and plan resource allocation more efficiently.

Charts and county info

Another feature of our final website is the ability to view charts regarding the number of covid cases in a certain county over time. When the user clicks on a county on the choropleth map, some information regarding the county, such as the covid community level, infection and vaccination rates, and the number of hospital beds available, are shown on the right side. A line chart showing the number of covid cases in the county is also shown. By having this information easily available, policymakers will find it easier to make planning decisions.

Preprocessing data

Many of the features of our website, such as the choropleth map and charts, require taking data from multiple sources, putting them together, and mapping them to the correct counties. Some values, such as the covid community risk level, also need to be calculated by taking data from multiple sources. In order to preprocess all this data, one of our team members created a Jupyter notebook, which when run, merges data taken from CovidActNow and CovidCareMap, filters out only the data required, calculates the risk level for each county, then maps all this data to their corresponding counties via a unique FIPS code. The result is outputted as a JSON file, which can then be easily used by different components in our website.

Problems encountered

Backend

One issue we faced was that we took a long time to understand what exactly was required from our API. Initially, we believed that all we needed to do was to get reports of diseases from the data source provided (Global Incident Map). As such, a majority of our work on the API was based around solely scraping and processing the data from the data source. However, after discussion with our mentor, we realised that we were actually required to get links to the articles regarding diseases from our data source, then analyse each article to generate a report for each disease mentioned in the article. Each report would contain details such as the time period of each case, symptoms, and location of cases. This was much more work than we expected, and we only had about a week to fix our API to match the requirements.

Due to this lack of time as well as lack of experience with scraping and processing data to match the requirements, the best we were able to do before the deadline was to get the URL, headline, date of publication, and main text of each article. For the reports section of each article, we were only able to generate one report object which contained a list of all the diseases and locations mentioned in the article, as well as a time period for all the cases. However, it was not fully completed to our satisfaction, for example, the time period would sometimes be longer compared to the actual time period mentioned in the article because only the smallest and largest mentioned dates would be extracted from the article.

Another issue we faced was scraping data. No one in our team had experience with scraping data from websites, so the members who were assigned to work on the scraper had to spend a significant amount of time learning how to scrape data. After they had gotten comfortable scraping from the data source, our team found that we would also need to scrape and analyse every single article. This was complicated, as each article website had different structures, making it hard to successfully scrape and analyse each article, compared to scraping from one website, like our data source. As such, it was almost impossible to successfully scrape and process each article. In the end, we had to resort to using a few external libraries to scrape and process each article, but even then there were a few instances where this wasn't fully successful.

We also faced many difficulties trying to deploy our API. We had decided to deploy our API using Heroku, as it was quick and simple, and one member had experience with Heroku. However, we quickly found that it wasn’t as simple as we had imagined. Because of the various tools and libraries we were using, our API was bigger than the size limit allowed by the free tier. One of our team members had to spend a few days trying to reduce the size of our API, which was no easy task as every single library we had installed was essential to our API.

Even after we finally managed to deploy our API, we still faced issues, such as our API timing out due to how long it took to get a response. The way our API worked, it would scrape article URLs from the data source based on the provided parameters, then scrape and process each article URL, before returning the results as a JSON file. Our data source is a very old website that takes a long time to fully load, so most of the time was wasted waiting for the pages of the data source to load, which would lead to the API exceeding the Heroku request timeout limit of 30 seconds. We did manage to fix this in the end, however it resulted in only part of the expected data to be returned. For example, if a user requested articles that mentioned the words ‘outbreak’ and ‘hantavirus’, the deployed API would only return articles that mentioned ‘outbreak’ to stay within the request timeout limit.

Frontend

The main issue we faced with our frontend was selecting a target user. We had a very vague idea of what our website should do, but weren’t sure who our target user should be, so instead we focused mainly on the features of our website, putting quantity over quality. Some of the features our initial frontend aimed to have included being able to view case reports around the world as a table, bookmark diseases and locations to receive alerts, and view various charts and predictions. However, due to the number of features we planned to have and their complexity, in the end, we were unable to properly implement most of the features. After our first demo, we received alot of feedback regarding the fact that our target user wasn't well defined, our frontend did not match the needs of our target user, and we had many features but none were well implemented. We were encouraged to try defining our target user and focusing on providing one or two key features very well, so for our final demo we ended up restarting our frontend almost completely from scratch, and transforming it into a single page app instead of a multipage website. As such, alot of the time and energy we had spent making various components and pages for the frontend felt like a waste.

Because of this overhaul of our frontend, we also had to spend more time learning to use new libraries and tools, which was hard due to the limited time we had. We would keep running into issues and errors with the libraries we used, and for some of them, such as react-simple-maps for drawing the choropleth map, and d3 for the colour legend scale, it was hard to debug due to the lack of documentation and resources.

Another issue we faced was regarding preprocessing the data we collected from various sources. Alot of the time our choropleth map and charts did not match the ones shown on our sources, even though they used the same data. It took a long time to debug and fix the Jupyter notebook that did the data preprocessing to match the original sources.

Skills we wish we had beforehand

One skill we wish we had before starting this project is how to scrape data from websites. None of us had any experience with making scrapers, so those who were allocated to creating the scraper had to spend a lot of time learning how to make a scraper, researching libraries and tools to use, and debugging errors. What would be especially useful is to learn how to scrape data no matter the type of website being scraped from, as the report creation part of the API required us to scrape and parse articles from various websites with different structures.

Another useful skill would be how to deploy our project. Only one of our team members was experienced with deployment using Heroku, and even then it took a long time for our deployed API to be functional, due to the many bugs and issues we faced along the way. All this time spent fixing these deployment issues could have instead been used to refine our API further to meet the requirements.

We would also have liked to have a bit more explanation on how to use Jira and Confluence beforehand. Due to courses such as COMP1531, most of us were used to writing and assigning tasks and issues on the same platforms where our code was contained, such as GitHub or GitLab. However, since Jira is seperate to GitHub, it was a bit confusing to use, so many of the sprints we conducted weren’t well documented on Jira.

What we would do differently

If we were to do the project all over again, we would firstly set stricter deadlines. The beginning of the project mainly involved planning and documentation, so we underestimated the scale of the project and the time needed to complete everything. This, coupled with not setting strict deadlines, resulted in us scrambling to complete tasks before deliverables and demos, as well as a lot of half-finished features.

For the API, our team understood the requirements of the API differently compared to what was actually required, and by the time we had realised our mistake, we didn't have much time to fully fix our API to match the requirements. If we had the chance, we would spend more time asking questions regarding the requirements of the API to avoid this mistake and spend more time working on the API so it worked properly.

For the frontend, we would spend a lot more time defining our target user and building our frontend features around their needs. Initially, we only had a vague idea of who our target user was, and were more focused on providing as many features as possible instead of how useful these features would be for our target user. Additionally, we were aiming to provide novel features that similar websites didn't offer without considering how feasible they were with the time we had and our current knowledge and skillset. Due to these factors, our frontend for the first demo had many features, but many of these were only half-working, and none of these features were useful for our target user. Although we did correct this for our final demonstration, if we had narrowed down our target user and their needs in the first place, we would have had more time to create a more polished final product.

Since we only had about two weeks between our first demo and final demo, we were only able to get the basics of our planned features working. For example, we had planned for the choropleth map to display data from anywhere around the world, especially data from developing countries. However, we were only able to get the choropleth map to display data for the US, due to time constraints and the data available. We were also only able to display data regarding covid, even though our initial plan was to display data regarding any outbreak. If we had a well-defined target user in the first place, we may have had the opportunity to implement these features for our final demo.

One more thing we would have done differently is explicitly assigning tasks to team members based on their skills and what they were comfortable with. Most of the time tasks were allocated based on what each member wanted to do, even if they had no experience to do the task. This often resulted in other team members needing to help out on top of their own responsibilities, and frequently we would end up swapping tasks. Additionally, the workload for some tasks was much more compared to other tasks, which would mean other team members would need to assist if they could in order to get everything done.