WorldPulse is a web application featuring an interactive 3D globe for monitoring global crises, disasters, natural hazards, and ongoing events in real-time. The application aggregates data from multiple authoritative sources (GDACS, USGS, ThinkHazard!, ACLED, UCDP, CrisisWatch, ReliefWeb) and displays them on a Mapbox-powered globe interface with enterprise-grade visualization capabilities.
The application provides users with the ability to explore the globe, search for locations, filter events by type and severity, and view detailed information about each disaster or crisis event. WorldPulse also features:
- Real-time news aggregation from 40+ global news sources via RSS/JSON feeds
- AI-powered location research reports using Valyu DeepResearch
- Subscription-based premium features with flexible pricing plans
- Conflict event tracking from ACLED and UCDP databases
- Image extraction for news articles
Technology Stack:
- React with TypeScript for component-based UI
- Wouter for client-side routing
- TanStack Query (React Query) for data fetching and caching
- Mapbox GL JS with globe projection for 3D visualization
- Shadcn UI component library built on Radix UI primitives
- Tailwind CSS for styling with custom design system
- Supabase Auth for authentication (Google, GitHub, email/password)
Key UI Components:
- Full-screen globe as the primary interface
- Fixed header with search and theme controls
- Collapsible left sidebar for map controls and filters
- Sliding right panel for event details
- Event markers with severity-based color coding
State Management:
- React Query for server state and caching (60-second refetch interval)
- Local React state for UI controls (sidebar visibility, selected events, filters)
- Supabase Auth state for user authentication
Technology Stack:
- Node.js with Express.js for REST API server
- TypeScript throughout the codebase
- Drizzle ORM for type-safe database operations
- Supabase for authentication and database
- Development mode uses Vite middleware for HMR
- Production mode serves pre-built static assets
API Design:
- RESTful endpoints following resource-based patterns
/api/events- Get all active events/api/events/:id- Get specific event details/api/events/type/:type- Filter events by type/api/config- Application configuration (Mapbox token)/api/auth/me- Get current authenticated user/api/news/feeds- Get all news feeds, create custom feeds/api/news- Get news articles with feed information/api/news/search- Search news with Valyu AI integration/api/subscription/checkout- Create subscription checkout session/api/subscription/webhook- Handle Polar webhooks/api/research/location- Start location research, get reports
Authentication:
- Supabase Auth with JWT-based stateless authentication
- Supports Google OAuth, GitHub OAuth, and email/password
- Backend validates JWT tokens from Authorization header
Scheduled Tasks: The application uses node-cron for scheduled data fetching and maintenance:
-
Event Data Fetching (every 6 minutes):
- GDACS natural disaster feeds
- USGS earthquake data
- UCDP conflict events
- ACLED conflict events
- CrisisWatch RSS feeds
- ReliefWeb humanitarian updates
- Automatic cleanup of events older than 7 days
-
News Feed Fetching (every 5 minutes):
- Fetches from all enabled RSS/JSON feeds
- Processes 40+ default feeds plus custom user feeds
- Image extraction and location geocoding
- Event deduplication and storage
-
Research Report Cleanup (every hour):
- Removes expired research reports (older than 5 days)
- Frees up database storage
-
Automatic Initialization:
- News feeds are automatically initialized on first run
- Default feeds are created if none exist in database
Data Processing:
- Scheduled data fetching using node-cron:
- Event fetching every 6 minutes (GDACS, USGS, UCDP, ACLED, CrisisWatch, ReliefWeb)
- News feed fetching every 5 minutes
- Research report cleanup every hour
- XML/RSS parsing for GDACS feeds and news sources
- JSON parsing for USGS earthquake data and JSON Feed format
- News feed aggregation with automatic initialization of 40+ default feeds
- Image extraction from articles with multiple fallback strategies (RSS images, Open Graph, Twitter cards, JSON-LD, HTML parsing)
- Location extraction from news text using geocoding
- Event deduplication using SHA256 hashing of normalized URLs
- Feed-specific event filtering and enable/disable functionality
- Automatic cleanup of old inactive events and expired research reports
Database:
- PostgreSQL via Supabase
- Drizzle ORM for type-safe database operations
Schema Design:
Events Table:
- Stores disaster events from multiple sources (GDACS, USGS, ThinkHazard!)
- Geospatial data (latitude/longitude) for map plotting
- Severity levels (green/yellow/orange/red) for visual prioritization
- Temporal data (start date, end date, last update)
- Metadata (affected population, magnitude, region, country)
- Indexed on source, event type, severity, and active status for query performance
Hazard Types Table:
- Reference data for ThinkHazard! categories
- Stores display metadata (icons, colors) for UI rendering
Users Table:
- id: Primary key (UUID)
- supabaseId: Supabase Auth user ID
- email: Normalized to lowercase for consistent lookups
- username: Display name
- password: Hashed password (for legacy email/password users)
- firstName, lastName, profileImageUrl: Profile data from OAuth
- subscriptionTier: 'free', 'premium', or 'pay_per_use'
- subscriptionStatus: Active subscription status
- polarCustomerId: Polar customer ID for subscription management
- hasUsedTrial: Whether user has used the 3-day free trial
- hazardLookupCount, newsLookupCount: Usage tracking for premium features
News Feeds Table:
- Stores RSS/JSON feed definitions for news aggregation
- Fields: name, url, type (rss/json), country, category, isDefault, enabled, createdBy
- Supports custom user-created feeds
- Automatic initialization of 40+ default feeds from major news sources
Location Reports Table:
- Stores AI-powered research reports generated via Valyu DeepResearch
- Fields: deepresearchId, locationName, locationLat, locationLng, countryCode, customQuery
- Report content stored as markdown with sources, wordCount, sourceCount
- Reports expire after 5 days (automatic cleanup)
- Status tracking: queued, running, completed, failed
Fetch Logs Table:
- Tracks data source reliability and fetch history
- Records success/failure status and error messages
# Supabase Configuration
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_ROLE_KEY=your-service-role-key
# Database (Supabase Postgres connection string)
DATABASE_URL=postgresql://postgres:password@db.your-project.supabase.co:5432/postgres
# Mapbox
MAPBOX_ACCESS_TOKEN=your-mapbox-token
# Polar Subscription Management
POLAR_ACCESS_TOKEN=your-polar-access-token
POLAR_WEBHOOK_SECRET=your-polar-webhook-secret
POLAR_SUBSCRIPTION_PRODUCT_ID=prod_xxxxxxxxxxxxxxxxxxxx
POLAR_PAY_PER_USE_PRODUCT_ID=prod_xxxxxxxxxxxxxxxxxxxx
APP_URL=https://your-domain.com # For checkout redirects
# Conflict Data Sources
ACLED_USERNAME=your-acled-email
ACLED_PASSWORD=your-acled-password
# Optional: ACLED_ACCESS_TOKEN=your-manual-token # If not using OAuth credentials
# News APIs
VALYU_API_KEY=your-valyu-api-key # For news search and location research
NEWSDATA_API_KEY=your-newsdata-api-key # Optional: NewsData.io API
MEDIASTACK_API_KEY=your-mediastack-api-key # Optional: Mediastack API
# Optional
GROQ_API_KEY=your-groq-api-key # For news summary generation
SESSION_SECRET=your-session-secret # For legacy session support
PORT=5000 # Server port (default: 5000)
NODE_ENV=development # or 'production'Create a .env file in the project root with these variables prefixed with VITE_:
VITE_SUPABASE_URL=https://your-project.supabase.co
VITE_SUPABASE_ANON_KEY=your-anon-key- Node.js 18+
- A Supabase project with:
- Google OAuth provider enabled
- GitHub OAuth provider enabled
- Email/password auth enabled
- A Mapbox account with API token
-
Clone the repository
-
Install dependencies:
npm install
-
Set up environment variables (see above)
-
Push the database schema:
npm run db:push
-
Start the development server:
npm run dev
npm run build
npm startWorldPulse includes a comprehensive RSS/JSON news feed aggregation system that automatically collects and processes news articles from major global sources.
Feed Support:
- RSS 2.0, Atom, and JSON Feed format support
- 40+ default feeds from major news sources (BBC, CNN, Reuters, NYT, Al Jazeera, DW, Guardian, Sky News, NPR, ABC News, CBS News, and more)
- Custom feed management: users can add their own RSS/JSON feeds
- Feed filtering: enable/disable specific feeds per user
- Automatic feed initialization on first run
Image Extraction: The system uses multiple fallback strategies to extract images from news articles:
- RSS-provided images: Media RSS (
media:content,media:thumbnail) and enclosure tags - Open Graph meta tags: Standard
og:imageproperty - Twitter card images:
twitter:imagemeta tags - JSON-LD structured data: Image URLs from structured data
- HTML parsing: First article image from HTML content (with semantic element prioritization)
- 8-second timeout for article fetching
- Automatic handling of Google News redirects
- Image URL validation and normalization
Location Extraction:
- Automatic geocoding of locations mentioned in article titles and descriptions
- Country-level fallback when specific coordinates aren't found
- Context-aware location extraction (uses feed country as context when available)
- Events without valid locations are stored with country information only
News Feed Panel:
- UI component for browsing and filtering news articles
- Feed-specific filtering and search
- Integration with main event display system
WorldPulse uses Polar for subscription and billing management, offering flexible pricing plans for premium features.
Subscription Plans:
Premium Plan ($10/month):
- 100 Valyu searches per month
- 3-day free trial for new users
- Full access to all features including:
- Map style selection
- PDF exports of research reports
- Priority support
- Unlimited news feed access
Pay-as-You-Go Plan ($0.10 per search):
- No subscription required
- Pay only for Valyu searches used
- Full access to all features
- Ideal for occasional users
Features:
- Polar checkout integration for seamless payment processing
- Webhook handling for subscription events (created, updated, active, canceled, revoked)
- Usage tracking:
hazardLookupCountandnewsLookupCountfor premium feature limits - Subscription tier management: free, premium, pay_per_use
- Trial management: tracks whether users have used their free trial
Integration:
- Backend validates subscription status for premium features
- Frontend displays subscription status and upgrade prompts
- Automatic subscription sync via Polar webhooks
WorldPulse includes an AI-powered location research system that generates comprehensive research reports about any location on the globe.
Valyu DeepResearch Integration:
- Uses Valyu DeepResearch API for deep, multi-source research
- Generates comprehensive markdown reports with citations
- Includes location context (nearby hazards, events, risk factors)
- Custom query support for targeted research
Research Process:
- User selects a location on the globe or searches for a location
- System gathers location context (hazards, nearby events, country information)
- Research task is queued and processed asynchronously
- Progress tracking with real-time status updates
- Report generation with markdown formatting
- Sources and citations included in report
Report Features:
- Markdown-formatted content with proper structure
- Source citations with URLs
- Word count and source count statistics
- PDF export functionality (browser-based print to PDF)
- Report storage with 5-day expiration (automatic cleanup)
- Report viewer modal for easy access
Usage:
- Premium users: Included in subscription (counts toward monthly limit)
- Pay-as-you-go users: $0.10 per research report
- Free users: Limited access (upgrade prompts)
Supabase:
- Authentication (Google, GitHub, email/password)
- PostgreSQL database hosting
- Row Level Security (optional)
Mapbox:
- GL JS library for 3D globe rendering
- Geocoding API for location search
- Multiple base map styles (satellite, streets, outdoors, dark, light, navigation)
GDACS (Global Disaster Alert and Coordination System):
- RSS/XML feeds for natural disasters
- Multiple feed endpoints (24h, 7d, 3m windows)
- Event-specific feeds (earthquakes, cyclones, floods)
- Provides severity alerts and affected population estimates
USGS (United States Geological Survey):
- GeoJSON API for real-time earthquake data
- Multiple magnitude thresholds (M1.0+, M2.5+, M4.5+)
- Time-based feeds (past hour, day, week, month)
- Provides detailed earthquake metadata (depth, magnitude type, felt reports)
ThinkHazard! (World Bank):
- Static hazard category reference data
- 11 hazard types: river flood, urban flood, coastal flood, earthquake, landslide, tsunami, volcano, cyclone, water scarcity, extreme heat, wildfire
- Used for categorization and UI reference
ACLED (Armed Conflict Location & Event Data Project):
- Real-time conflict events: battles, protests, riots, political violence
- Geo-located events with detailed metadata (actors, fatalities, source)
- OAuth-based authentication (free for non-commercial use)
- Provides granular incident mapping and violence timelines
UCDP (Uppsala Conflict Data Program):
- Organized violence and conflict data
- Both event-level (geo-located) and conflict-level (aggregated) datasets
- Public API with no authentication required
- Long-term quantitative conflict analysis
CrisisWatch (International Crisis Group):
- Expert-curated crisis monitoring via RSS feeds
- Country-specific and regional feeds
- Qualitative monitoring with status and trend indicators
- Provides high-level conflict status and summaries
ReliefWeb:
- Humanitarian updates and crisis information
- RSS feeds for humanitarian emergencies
- Maps and situation reports
- Automatic sync of active events
News Feeds (RSS/JSON):
- 40+ default feeds from major global news sources
- Sources include: BBC, CNN, Reuters, NYT, Al Jazeera, DW, Guardian, Sky News, NPR, ABC News, CBS News, and many more
- Support for RSS 2.0, Atom, and JSON Feed formats
- Custom feed management (users can add their own feeds)
- Automatic location extraction from article text
- Multi-strategy image extraction from articles
MIT