Koa.js - Streaks API

Docker, MongoDB, TypeScript, Koa.js (Node.js), Jest testing
https://andromeda.technology

Koa.js - Streaks API

1. Technology

We use MongoDB and Mongoose, to allow us to evolve database design with ease, but we, at the same time, also use defined Schemas, and strict-evaluation of inputs and outputs via TypeScript interfaces, to provide clarity and stability to the code and support future changes.

Environment variables are used and parsed via Dotenv and Joi.

Integration and Unit Testing is done via Jest, SuperTest, MongoDBMemoryServer.

Docker provides a MongoDB container for the app.

• TypeScript,
• Koa.js,
• Database: MongoDB: Mongoose,
• Config: Dotenv, Joi,
• Testing: Jest: SuperTest, MongoDBMemoryServer,
• Docker: MongoDB.

2. Usage

1. Clone the repo,
2. Duplicate .env.example files in [./,/docker/] to .env; modify as needed,
3. Have Docker installed,
4. Run the Docker containers and your app (check the instructions below),
5. Run the tests.

3. Setup

Docker

Docker provides isolated MongoDB for your project.

cd ./docker

# Duplicate example env file, modify as needed
cp .env.example .env

# Docker start
docker-compose up -d
# Or use the following:
# docker compose up -d

Application

# Return from `docker` to root dir
# cd ..

# Duplicate example env file, modify if needed
cp .env.example .env

# Install packages
npm i

# Run
npm run dev

Application .env variables

Check and adjust variables in, previously duplicated, .env file. Check instructions above.

ACTIVITY_THRESHOLD=30

4. Modules

1. User Module,
2. Activity Module,
3. Streak Module,
4. Leaderboard Module,

All API routes are prefixed by API_PREFIX (defined in.env) (default: /api).

5. Features - details, reasoning, concerns, future solutions

Purpose and implementation

• We save User with timezone
• We save Activity with { user_id, date_iso_start, date_iso_end }
• We use combination of User and Activity to calculate Streaks and Leaderboards

NOTE: for now we treat every activity as UTC, for ease of use.

ACTIVITY threshold

Currently in minutes. Another variable can be added to change type, e.g. to seconds, hours. It all depends on the app needs.

ACTIVITY CRUD actions - impact on CACHING and solutions

We can cache calculated streaks, per day. But this all depends on the actions allowed on the data, e.g. editing, deleting saved Activity data afterwards... As then, that cache needs to be invalidated, cleared, and re-calculated.

For this project, we just save Activity, allow all CRUD actions, at any time... As we don't store cached values. We calculate streaks and leaderboards on-the-fly.

For more advanced usage, we need specified use-cases, as this impacts storing, caching, retrieving strategy for all mentioned data.

Leaderboard - Multi-zone concerns and solutions

We calculate global leaderboard ONLY. Later, userUser models can be introduced, to support relationships between users.

When combining multi-user activity into leaderboard, we have potential multi-zone differences. For now, we use UTC timezone. Later, we can use initial user timezone, and adjust other user's activity into this timeframe, depending on the needs of the app.

Timezone: Current and future implementation

We currently save Timezone as string in form of e.g. "America/New_York", and this data is sent as a part of the request body (in Axios, XHR request, or Postman request), which is ok for this phase.

Another solution: Timezone can also be saved as { is_positive: [true,false], offset: [NUMBER] }.

Timezone can be defined as a separate entity/model, and referenced via ID. This simplifies input, output, testing on backend and frontend. This timezone ID can be used in User.update request, instead of sending full name of the timezone, which we currently use.

Moment.js

• Timezones introduce complexity that needs to be handled properly
• Moment.js, or similiar library, can be used to offload handling logic for timezones, depending on the app and current state

Timezone normalization

• Services must normalize start,end dates into nullified timezone - to be comparable, all to UTC or other

Streak calculation

Current implementation

• StreakHelper, UserService, ActivityService, DateHelper - modular approach to handling per User, per Date, multi-day spanning activity streaks
• Is a good balance of data retrieval and processing time - while being correct
• Takes any Date as input (current, in the past, or in the future)
• Keeps querying for activity starting from currentDate, into the past
• Takes single, or multiple spanning days, from activity, as a streak
• Continues fetching, from furthest day in the current activity timeframe (thus optimizing querying)
• Stops fetching when current day does not satisfy .env requirements (THRESHOLD variable)
• Check integration tests in streak.helper.spec.ts

Further optimization:

• Caching, while saving Activity data; not to re-calculate every time, on-the-fly
• Duplicated data is skipped during calculation, it can also be skipped during input.

Integration tests coverage

For integration tests, please check streak.helper.spec.ts.

• multi day spanning activity streaks (single activity starting and ending on different days)
• no streak days (for any specified date)
• streaks from the past (at specific date in the past)

Future steps

• User referencing, on Database level (e.g. Activity.user_id -> User.id); Now only in Logical layer

• Caching per user, per day - in MongoDB, Redis or 3rd DB, depending on the app

• Activity.service.findAll can have DTO named properly, to address input parameter(s) purpose or a new Service can be created, on a higher level

• Streak.helper.find - mind the usage of setUTCHours vs setHours

5. Tests

Using Jest Testing Framework.

Jest uses SuperTest and MongoDBMemoryServer.

npm run test

6. Postman

• Postman Documentation

Pre-set environment variables:

• host
• admin_password

Dynamic environment variables, automatically set in tests:

• access_token

7. Admin Routes

Routes can be protected with jwtCheck middleware, requiring admin rights.

Requests going to these routes require Authorization: Bearer {access_token} header.

List of protected, i.e. Admin Routes

1. Message[Create,Update,Delete],
2. [Add your protected routes here]

Getting access_token for the Admin user

• Request endpoint: POST /auth/token,
• Pass your password in the request body: { password: ADMIN_PASSWORD },
• Response will return created token.

Note: Postman collection will automatically set access_token environment variable, so you can immediately call admin routes, without copy-pasting it or setting the env variable manually.

Getting the ADMIN_PASSWORD

• Your ADMIN_PASSWORD is defined in .env file.
• It defaults to secret.

8. Deployment

If you use MongoDB Atlas: Uncomment and fill DB_URI in .env.

9. Social

Andromeda

• Medium
• Twitter

10. Rest

Hero image source: FireStarter, gilad, DevianArt.

11. Related

🔥 FireStarter API

• Firestarter API - Progressive Startup API Boilerplate

🏄 Habitus

• State-of-the-art tracker for emotions, habits and thoughts,
• Healthiest version of you,
• Gamified,
• Anonymous and open source.

12. Contribute

Check Self-Aware Software Artisan before contributing.

Started in Europe, Croatia 🇭🇷

Crafted with ❤️
by contributors around the 🌍 World and 🌌 Andromeda.