TODO:
Links to final source code (github repo) Mashira Farid
https://github.com/samztz/SENG3011_GeeksForHDs
Use cases/ requirements of API (Focus on what has been achieved)Zifan Wei
The system design and Implementation
- update from feedback Avijit Prasad
- Final software architecture (diagram) Tingzhuang Zhou
- tech stack we using (any changes) Tingzhuang Zhou
- other teams API using (used team minions' API but discard finally- justify this) Avijit Prasad
- existing API using Avijit Prasad
- Summary of Key benefits/achievements of project relating to design/implementation Lin Thit Myat Hsu
Team organization and conclusion/appraisal of your work
- Responsibilities of each member Lin Thit Myat Hsu (Anyone else can help me figure out who did what it’d be perfect)
- How did the project go Mashira Farid
- Major achievements in project
- Issues/problems encountered
- What kind of skills you wish you had before the workshop (this way we can try including them in other courses)
- Would you do it any differently now? • I.e. tools, different technology, time management, et
EPIDENCE
...
TEAM
Tingzhuang Zhou Avijit PrasadMashira Farid Zifan Wei Lin Thit Myat Hsu
APIs
...
https://covidactnow.org/data-api
Risk metric is received which is displayed on the risks maps. Risk metric is determined by population of the county, weekly infection rate, staffed hospital beds and ICU beds in the county and scaled in a range from 1 to 5.
Vaccination ratio is received by counties which is later converted into vaccination percentage and displayed with counties.
Weekly covid cases by counties are received which is used to determine community threats in each county.
...
NY TIMES API
New York times API gives county wise data for charts to be displayed in each county mapping showing number of cases with time.
https://github.com/covidcaremap/covid19-healthsystemcapacity/tree/master/data/published
...
...
EPIDENCE
...
TEAM
Tingzhuang Zhou Avijit PrasadMashira Farid Zifan Wei Lin Thit Myat Hsu
APIs
Note :- To see the interaction with the frontend look at Software Architecture Figure 2
https://covidactnow.org/data-api
Risk metric is received which is displayed on the risks maps. Risk metric is determined by population of the county, weekly infection rate, staffed hospital beds and ICU beds in the county and scaled in a range from 1 to 5.
Vaccination ratio is received by counties which is later converted into vaccination percentage and displayed with counties.
Weekly covid cases by counties are received which is used to determine community threats in each county.
https://github.com/covidcaremap/covid19-healthsystemcapacity/tree/master/data/published
Following dataset is used to receive hospital beds data which is later added to risk data. All the calculations and preprocessing done in. https://github.com/samztz/SENG3011_GeeksForHDs/blob/main/PHASE_2/epi_frontend/data/data_preprocessing.ipynb which produces final data in this format:
fips | risk | infectionRate | county |
|
|
|
|---|---|---|---|---|---|---|
https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties-recent.csv
New-York times API used to populate charts in every county
Data
Data is available at Data folder in github.
To get final preprocessed data https://github.com/samztz/SENG3011_GeeksForHDs/blob/main/PHASE_2/epi_frontend/data/
...
...
Feedback
Deliverable 1
Application Architecture and Tech stack
By the time goes by, the software architecture is constantly changing over time. The change base on the iterating requirements, changing end-users, and also the UI design.
Initially, we wanted to achieve features like user authentication, bookmark of queries, subscription, warnings, data visualizations in list/map/charts, case predictions. And we are targeting the end user of data analyst. This initial software architecture be like:
...
We have our node.js backend connected MongoDB atlas for features like user authentication, bookmarks feature, and we have the NextJS/React/Redux/MUI front end to build the client side applications.
…
Backend - NodeJS, ExpressJS
NodeJS: an open-source, cross-platform runtime environment for JavaScript built on Google’s V8 engine
We chose to use NodeJS as it is very simple to use and easy to learn.
Since Node is event-driven, it can also enable asynchronous programming, which provides great performance compared to Python
Since NodeJS is open-source and cross-platform, everyone in the team could develop the app regardless of their development environment
By using NodeJS, we can also use NPM (node package manager), which allows us to install numerous libraries to use in our code to help in development. ExpressJS is an example of one of these libraries which is installed using NPM
ExpressJS: a backend web application framework that's used to design and build web apps quickly and easily
We decided to use Express as it allows us to create RESTful routes easily. This is particularly useful as we will be developing our own API as part of the project, which we will need to create custom routes for
Since Express is an very popular NPM library, it works well with Node
Express also supports JSON, which we plan on using for our API to return results
Frontend - ReactJS
ReactJS: an open-source frontend JavaScript library
We chose ReactJS for the frontend because it supports the use of components, which allows us to reuse existing code instead of having to rewrite similar features, which in turn allows for faster development
React is also better in terms of performance compared to other frameworks like Angular, as it only updates components in the DOM whose state has changed, instead of updating every component again, making our app lightweight and flexible
React also has plenty of documentation and online resources, so the members of our team who were unfamiliar with it would be able to pick it up quickly
Development and Deployment Environment:
Development - Cross-Platform
Since everyone in our team is using different operating systems on our personal machines, we ensured that all the technologies we chose as part of our tech stack worked across different platforms
This allows everyone in our team to work on our app in the development environment they were most comfortable with and had access to, which in turn leads to more efficient development of our app, compared to setting a specific development environment that not everyone may have access to
Deployment - Heroku (later switch to AWS if time permits)
Heroku: a cloud platform as a service for app deployment
We chose to deploy our API using Heroku as it only requires a few steps in order for deployment. This will make it quicker and easier to deploy our API compared to other platforms such as AWS
Additionally, since our API is small, it doesn’t need all the functions that AWS provides, making it much simpler for us to deploy it using Heroku
However, if time permits, we plan on deploying our API using AWS
Database - MongoDB
MongoDB: a cross-platform, NoSQL database
Initial database options are: MongoDB, PostgreSQL, SQLite
We chose MongoDB as our database as our app isn’t very big, so we didn't need to use SQL
MongoDB is more scable as it will initial a new data fields whenever we create the documents in database, hence suitable for agile iteration when adding new feature later.
Additionally, MongoDB integrates well with NodeJS, and stores data in JSON format, which matches the format our API will output in
Testing - Postman
Postman: API testing application
We chose Postman over other tools such as Insomnia to test our API as a few of our members had experience using it
Postman also provides the ability to collaborate as a team to write, store and run test cases
Specific Libraries:
Library - PuppeteerJS, Morgan(logger), Winston(Logger), Nodemon, CORS (TODO)
PuppeteerJS: an open-source NodeJS library
We use PuppeteerJS for scraping the website as the library allows the backend to run a headless (a code-only) version of Chrome to load the websites on.
A headless version of Chrome doesn’t use as much resources and as such can be easily run in any server, such as the hosting servers that we are using.
PuppeteerJS also have high-level API control of the Headless Chrome browser via the DevTools Protocol. This allows for the code to read the structure code of the website directly and to access the required parts of the website.
Morgan: HTTP request logger middleware for Node.js
Winston: Another logger for both develop and production environment.
CORS: NPM package for solving Cross Origin Resource Sharing problems
dotEnv: abstracting environment variable and url configurations in .env file, making swaping of environment between development/production feasible.
UI design - Material UI
Material UI (MUI): an open-source frontend framework for React components based on Google’s Material Design
We decided to use MUI as it would make the development of our app quicker and easier. Since MUI defines its own components, we can use them to quickly design the UI of our app, instead of spending time writing our own CSS classes
Since we had decided to use React for our frontend, and MUI uses React components, it was only logical we used MUI over other non-React frameworks such as Bootstrap
Additionally, some of our team members already had some experience using it. The extensive documentation also means it will be quick and easy for the team members who haven’t used it before to pick up, as well as debug
By using MUI, we can keep our frontend UI theme consistent, and even customize it to suit our needs
Recharts JS - A composable charting library built on React components
We choose Richards over the other visualization kit, because the document is easy to read so we can easily implement the chart in our own
While testing the charts framework, we found out Recharts includes built in neat animation by default, the animation will further enhance user experience
Key Benefits
Our first main key benefit was the total overhaul of the design compared to our old week 7-8 design. While that design had some benefits, with help from our tutor we managed to refine our ideas and move from an application which did a lot of multiple things decently to a much better application that did one thing very well - all done within only 2 weeks.
Another benefit we found is the singe-page design of the website. Compared to other designs we’ve seen, a single-page design like the one we have reduced the load time of loading multiple websites and making the user experience much faster and smoother for the user to use.
The website’s map’s ability to highlight certain information also makes it easier for users to gather and display information in a quick and efficient manner compared to other websites, and since it’s all accessible on one page, switching between different maps is quick and easy. The fact that each of the areas on the map can be interacted to display as more information, as well as the graphs, makes our website a potent source of information at any given time.
Use cases:
The target user would be policy makers who are working in local authority in USA.
Our website could deliver visualisation graphs which are related to the analysis of pandemic risk level among all counties in USA and medical resources distribution.
There are four types of visualisation graphs provided to target user:
a. Risk Level.
b. Vaccinations.
c. Staffed All Beds [Per 1000 Adults (20+)].
d. Staffed ICU Beds [Per 1000 Adults (20+)].
e. Licensed All Beds [Per 1000 Adults (20+)].
When a policy maker checks the severity of one specific county, there is one customised colour bar with discrete number as indicator shown below the risk level graph. It can use colour to differentiate the severity among all counties
When a policy maker hovers on the specific county, there is one small black box pops up. It describes the risk level of that county in a simplified way.
When a policy maker clicks one specific county on the graph, there is one side bar on the right-hand side pops up, it contains three parts, first part above is including county name and risk level latest update date. Second part middle includes a status bar that shows the severity of that county based on the risk level. As well as infection rate and other related information. Third part at the bottom exists one line graph which records the change of the number of epidemic cases in past few days. This detailed information can bring a clear overview of the county during pandemic which gives policy maker makes appropriate decision.
When a policy maker checks the vaccinated rate of one specific county, the map type would be switched to Vaccinations graph. There is one customised colour bar from light colour to dark colour to indicate how large the vaccinated rate is. It uses colour to have a clear comparison of vaccinated rate between adjacent counties.
When a policy maker checks how many empty beds left, the map type would be switched to Staffed Beds graph. There is one customised colour bar from light colour to dark colour to indicate how many beds left in each county. It could make it possible for distributing medical power between counties because the complete data analysis of staffed beds and risk level provided in the visualisation. The policy makers can compare the risk level of the county they live in with their adjacent counties, as well as comparing the staffed beds left between them. Thus, it could show the possibility to borrow some beds from one county to another which dealing with the insufficient supply of beds for that county.
Responsibilities of Each Member
· Mashira was our main coder for the project, making up most of the backend work, which included the API connections and parsing data from the API to something that the frontend can use (usually involving converting it from JSON)
· Lin was one of the designers of the application, setting up and creating the front-end prototype of the website, as well as cleaning up and advising features that might have been either good or possibly unneeded. He also was the one who did the clean-up and Q&A of the program and the report.
· Avijit(?) was the one who worked on the API the most and created a new dataset and algorithm which allowed us to display all the information on the map.
· Zifan: doing some code about scrapping data from website and also creating two pages in front-end, and doing use case of the report.Through feedback we realised our understanding of article and report data that we were supposed to produce is not correct. We were also asked to document our project management better through Grantt charts and more detailed communications which we included in the final report
Deliverable 2
We were advised to test the APIs in a correct manner. We were told about the structure or articles and reports we were supposed to get which helped us in our implementation.
Deliverable 3
In the first Demo we realised our use case and requirements were not properly defined. We focussed on a broad range of users and our website had too many features that didn't cater to a specific target audience. Our main goal from the start was to create a website that caters to policy makers. To help our target user do resource management and surge planning with the aid of our website. We initially decided to make our websites for analysts as we thought we would provide them with different visual aids (charts and map) with some raw data to help them do their analysis better in outbreaks.
After the demo, we got feedback from our mentors that the idea to provide visual aids and data is good but the target audience does not require these services as they specialise on them. We took our time and through the feedback changed the whole website with a more clear target user. Adopting the age old 'Do not fall in love with your code' approach we started our project from ground up but this time with a more clear use case and service we were providing.
We removed redundant features and shifted our focus to policy makers in countries, specially developed countries, so they can get relevant data and visualisations to help them. We also got an idea to focus on small suburbs rather than countries to better represent our data from our mentor. Through this we focussed on making one map but made it as detailed as possible with enough insights so a policy maker can make most of his resource decisions through it.
We also preprocessed datas from different sources to give useful insights to the user. In the end, the feedback process helped us to understand the importance of having a Use Case in mind before implementing features.
Application Architecture and Tech stack
By the time goes by, the software architecture is constantly changing over time. The change base on the iterating requirements, changing end-users, and also the UI design.
Initially, we wanted to achieve features like user authentication, bookmark of queries, subscription, warnings, data visualizations in list/map/charts, case predictions. And we are targeting the end user of data analyst. This initial software architecture be like:
...
We have our node.js backend connected MongoDB atlas for features like user authentication, bookmarks feature, and we have the NextJS/React/Redux/MUI front end to build the client side applications.
...
But soon we realize it is very overwhelming for us to complete everything in demo, and we didn’t do well in week 8’s demo, therefore after we deeply reconsider the feedback from the demo, we simplified our application architecture from multipage application to simple page one and focusing on the user interaction of data that will benefits to customer. the reworked architecture is showing blow:
The architecture is more straight forward that we have our NextJS rendering the React simple Map, and the side information Card, the card contents data plus Recharts case chart, each time user click into different counties in the map, the Card will be filtering data from API calls by useState hooks.
Backend - NodeJS, ExpressJS
NodeJS: an open-source, cross-platform runtime environment for JavaScript built on Google’s V8 engine
We chose to use NodeJS as it is very simple to use and easy to learn.
Since Node is event-driven, it can also enable asynchronous programming, which provides great performance compared to Python
Since NodeJS is open-source and cross-platform, everyone in the team could develop the app regardless of their development environment
By using NodeJS, we can also use NPM (node package manager), which allows us to install numerous libraries to use in our code to help in development. ExpressJS is an example of one of these libraries which is installed using NPM
ExpressJS: a backend web application framework that's used to design and build web apps quickly and easily
We decided to use Express as it allows us to create RESTful routes easily. This is particularly useful as we will be developing our own API as part of the project, which we will need to create custom routes for
Since Express is an very popular NPM library, it works well with Node
Express also supports JSON, which we plan on using for our API to return results
Frontend - ReactJS
ReactJS: an open-source frontend JavaScript library
We chose ReactJS for the frontend because it supports the use of components, which allows us to reuse existing code instead of having to rewrite similar features, which in turn allows for faster development
React is also better in terms of performance compared to other frameworks like Angular, as it only updates components in the DOM whose state has changed, instead of updating every component again, making our app lightweight and flexible
React also has plenty of documentation and online resources, so the members of our team who were unfamiliar with it would be able to pick it up quickly
Development and Deployment Environment:
Development - Cross-Platform
Since everyone in our team is using different operating systems on our personal machines, we ensured that all the technologies we chose as part of our tech stack worked across different platforms
This allows everyone in our team to work on our app in the development environment they were most comfortable with and had access to, which in turn leads to more efficient development of our app, compared to setting a specific development environment that not everyone may have access to
Deployment - Heroku (later switch to AWS if time permits)
Heroku: a cloud platform as a service for app deployment
We chose to deploy our API using Heroku as it only requires a few steps in order for deployment. This will make it quicker and easier to deploy our API compared to other platforms such as AWS
Additionally, since our API is small, it doesn’t need all the functions that AWS provides, making it much simpler for us to deploy it using Heroku
However, if time permits, we plan on deploying our API using AWS
Database - MongoDB
MongoDB: a cross-platform, NoSQL database
Initial database options are: MongoDB, PostgreSQL, SQLite
We chose MongoDB as our database as our app isn’t very big, so we didn't need to use SQL
MongoDB is more scable as it will initial a new data fields whenever we create the documents in database, hence suitable for agile iteration when adding new feature later.
Additionally, MongoDB integrates well with NodeJS, and stores data in JSON format, which matches the format our API will output in
Testing - Postman
Postman: API testing application
We chose Postman over other tools such as Insomnia to test our API as a few of our members had experience using it
Postman also provides the ability to collaborate as a team to write, store and run test cases
Specific Libraries:
Library - PuppeteerJS, Morgan(logger), Winston(Logger), Nodemon, CORS (TODO)
PuppeteerJS: an open-source NodeJS library
We use PuppeteerJS for scraping the website as the library allows the backend to run a headless (a code-only) version of Chrome to load the websites on.
A headless version of Chrome doesn’t use as much resources and as such can be easily run in any server, such as the hosting servers that we are using.
PuppeteerJS also have high-level API control of the Headless Chrome browser via the DevTools Protocol. This allows for the code to read the structure code of the website directly and to access the required parts of the website.
Morgan: HTTP request logger middleware for Node.js
Winston: Another logger for both develop and production environment.
CORS: NPM package for solving Cross Origin Resource Sharing problems
dotEnv: abstracting environment variable and url configurations in .env file, making swaping of environment between development/production feasible.
UI design - Material UI
Material UI (MUI): an open-source frontend framework for React components based on Google’s Material Design
We decided to use MUI as it would make the development of our app quicker and easier. Since MUI defines its own components, we can use them to quickly design the UI of our app, instead of spending time writing our own CSS classes
Since we had decided to use React for our frontend, and MUI uses React components, it was only logical we used MUI over other non-React frameworks such as Bootstrap
Additionally, some of our team members already had some experience using it. The extensive documentation also means it will be quick and easy for the team members who haven’t used it before to pick up, as well as debug
By using MUI, we can keep our frontend UI theme consistent, and even customize it to suit our needs
Recharts JS - A composable charting library built on React components
We choose Richards over the other visualization kit, because the document is easy to read so we can easily implement the chart in our own
While testing the charts framework, we found out Recharts includes built in neat animation by default, the animation will further enhance user experience
React Simple Maps
Create beautiful SVG maps in react with d3-geo and topojson using a declarative api.
We choose the React Simple Maps as our core package, over the other maps like google maps, because it is a React map library that allow us to customize a zoomable, Composable map component, that help use develop the functionalities like click to switch county display.
Key Benefits
Our first main key benefit was the total overhaul of the design compared to our old week 7-8 design. While that design had some benefits, with help from our tutor we managed to refine our ideas and move from an application which did a lot of multiple things decently to a much better application that did one thing very well - all done within only 2 weeks.
Another benefit we found is the singe-page design of the website. Compared to other designs we’ve seen, a single-page design like the one we have reduced the load time of loading multiple websites and making the user experience much faster and smoother for the user to use.
The website’s map’s ability to highlight certain information also makes it easier for users to gather and display information in a quick and efficient manner compared to other websites, and since it’s all accessible on one page, switching between different maps is quick and easy. The fact that each of the areas on the map can be interacted to display as more information, as well as the graphs, makes our website a potent source of information at any given time.
Use cases:
We must flatten the curve to give our healthcare system more time to prepare for this surge of patients. Preparation means enough space, staff, supplies, and systems at the right places and times to meet patient needs. Through mapping and geospatial analytics, we provide situational awareness and foresight for health systems, policymakers, emergency managers, suppliers, and the public to plan and coordinate preparations between facilities, counties, states, or nationwide.
Use case 1:
A policy maker wants to know the risk level of his county in the surge of the pandemic to recognise any resource needs he needs to address
Requirements:
Visual mapping of risk on every county of USA
Displaying risk on every county
Requirements Fulfilled:
Risk data mapped to a map
When a policy maker checks the severity of one specific county, there is one customised colour bar with discrete number as indicator shown below the risk level graph. It can use colour to differentiate the severity among all counties
Policy maker can hover on map to display risk level
Limitations:
Problem:
Data not called dynamically from an API
Solution:
Host the python file in lambda function which in futures can be called to get processed data.
Use case 2:
Policy maker can also look into his county for more information which will give him insights of the risk being shown. For example he can look the vaccination rate of the county and increase vaccination programmes to solve the issue is the rate is less.
Requirements:
Letting the user click on the chart to zoom in the county which displays more metrics of the county
Requirements Fulfilled:
When a policy maker clicks one specific county on the graph, there is one side bar on the right-hand side pops up, it contains three parts, first part above is including county name and risk level latest update date. Second part middle includes a status bar that shows the severity of that county based on the risk level. As well as infection rate and other related information. Third part at the bottom exists one line graph which records the change of the number of epidemic cases in past few days. This detailed information can bring a clear overview of the county during pandemic which gives policy maker makes appropriate decision.
...
Use case 3:-
If a policy maker sees his county is in risk , he may want to get an idea of hospitals beds in his county to accommodate more patients in the coming future. That includes total beds, staffed beds and ICU beds to recognise where he should focus to manage his resources to minimise risk. These metrics are given with a ratio to population in the county. For example a county can have a high risk and sufficient beds, but it has less ICU beds which is seen through the charts. Also these charts can give insights about other counties so counties with high resources and less risk can be recognised by the user and the policy maker can communicate with them for resources
Requirements:-
User can change the map types to see different hospital beds ratio in different counties.
Hospital beds ratio mapped to counties with an appropriate colour scale
Requirements fulfilled:-
When a policy maker checks how many empty beds left, the map type would be switched to Staffed Beds graph. There is one customised colour bar from light colour to dark colour to indicate how many beds left in each county. It could make it possible for distributing the resource between counties because the complete data analysis of staffed beds and risk level provided in the visualisation. The policy makers can compare the risk level of the county they live in with their adjacent counties, as well as comparing the staffed beds left between them. Thus, it could show the possibility to borrow some beds from one county to another which dealing with the insufficient supply of beds for that county.
Limitations:-
Problem:
Data is processed from API’s but is not dynamically available
Solution:
Create lambda function of python file and receive data from hosting an API from it
Use case 4:-
If the policy maker has to do his own surge planning through tools available on different websites like
https://www.euro.who.int/en/health-topics/Health-systems/pages/strengthening-the-health-system-response-to-covid-19/surge-planning-tools/adaptt-surge-planning-support-tool they are able to download data in our website
Requirement:-
Provide link to download data to the user
Requirements fulfilled:-
When a policy maker first go to the page of our website, it would pop up a model which contains the da link and confluence link. Thus, they can download the data as they want.
...
Other Limitations for future updates:
Problem:
Users cannot be alerted with risk level changes.
Solution in future updates:
When risk level changes which can be achieved through a setInterval method in use-state module of React. Users get notified through email with EmailJS.
Future use case:
Our main goal of the project is to increase the maps to developing countries which needs tools like these the most. As we know in the previous pandemic countries like India had a problem with hospital beds and medical supplies. Also these tools focus on small areas as people managing those areas do not always have a data analyst to aid with these important projections. Visual maps makes those policy makers life easier to recognise in detail if they need to act fast with their resource management.
...
Responsibilities of Each Member
Roles | Member | Responsibilities Expected | Responsibilities Delivered | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Project manager |
|
| |||||||||||||||||||||||||
Frontend Developer |
|
| |||||||||||||||||||||||||
Backend Developer and Software Architect |
|
| |||||||||||||||||||||||||
Business Analyst |
| ||||||||||||||||||||||||||
Risk Analysis and Testing |
|
|
Unassigned Responsibilities (done by the whole team)
Project Setup
Github Repo Management
Presentation drafts
Research to refine Use cases
Finalising features of the system
Jira roadmap
https://unswseng.atlassian.net/jira/software/projects/SENG3011/boards/46/roadmap
Deliverable 1
Lin Thit Myat Hsu mapped out the report, and was tasked with keeping everyone up to date and coordination of resources, and as such mapped out the project plan.
Mashira Farid and Avijit Prasad were making plans for the API and the basic code, while Tingzhuang Zhou was tasked with justifying the language and development environments.
Deliverable 2
Mashira Farid made the scrapper
Lin Thit Myat Hsu made the tests
Tingzhuang Zhou made the APis and devided the tasks
Avijit Prasad helped with the scrapper
Zifan Wei helped with the scrapper
Deliverable 3
Avijit Prasad decided use cases and devised work, he also worked on services page that adhered to the use case
Mashira Farid made the UI/UX design and made Home page, Reports page for the demo
Tingzhuang Zhou did project setup and backend
Lin Thit Myat Hsu Did risk analysis and worked on landing page
Zifan Wei Refined use cases
Deliverable 4
Avijit Prasad finalised use case and allotted everyone tasks
Tingzhuang Zhou made the side bar
Mashira Farid made the maps and UI/UX decisions
Lin Thit Myat Hsu Made popup page and risk analysis
Zifan Wei Refined the final use cases and did requirement analysis
Project Summary
Major achievements
...