🚀 TalentSync System Architecture
Complete technical documentation for the AI-driven CV-to-RFP matching platform
System Overview
TalentSync is an Enterprise-grade AI-driven platform for matching consultant CVs to RFPs and job listings, featuring multi-tier queue management, multiple LLM providers, and automated job crawling.
Total Codebase
920K
Lines of Code
Database
115
PostgreSQL Tables
API Endpoints
200+
RESTful APIs
Queue Tiers
5
Processing Levels
Technology Stack
🔧 Backend
Node.js v18+ with Express.js
489,343 lines of JavaScript
Socket.io for real-time updates
JWT authentication
🗄️ Database
PostgreSQL 14+
115 tables, 303 indexes
Row Level Security (RLS)
Multi-tenant architecture
🎨 Frontend
HTML/CSS/JavaScript
341,955 lines total
PWA support
Mobile responsive
🤖 AI/ML
Multi-Provider Support
OpenRouter (Primary)
Ollama (Fallback)
OpenAI Compatible
🧪 Testing
K6 & Playwright
155,295 lines of tests
E2E automation
Load testing
🚀 Deployment
PM2 Process Manager
Multi-core utilization
Auto-restart
Load balancing
Key Features
- ✅ CV to RFP Matching with AI-driven analysis and scoring
- ✅ Automated Job Matching with continuous matching against new jobs
- ✅ Multi-language Support (13 languages with Swedish priority)
- ✅ Professional PDF Report Generation with visualizations
- ✅ Real-time Admin Dashboard with monitoring and analytics
- ✅ 8 Active Job Crawlers (Verama, Cinode, Keyman, Upgraded, etc.)
- ✅ CheatAI Mode for premium users (AI-undetectable content)
- ✅ GDPR Compliant with audit logging and data retention policies
System Architecture
High-Level Architecture
graph TB
subgraph "Frontend Layer"
UI[Web UI/PWA]
Mobile[Mobile App]
Admin[Admin Dashboard]
end
subgraph "API Gateway"
API[Express.js API]
WS[WebSocket Server]
Auth[JWT Auth]
end
subgraph "Business Logic"
QM[Queue Manager]
AI[AI Processor]
Crawler[Job Crawlers]
Match[Matching Engine]
end
subgraph "Data Layer"
PG[(PostgreSQL)]
Redis[(Redis Cache)]
Files[File Storage]
end
subgraph "External Services"
LLM[LLM Providers]
Email[Email Service]
SMS[SMS Gateway]
end
UI --> API
Mobile --> API
Admin --> API
Admin --> WS
API --> Auth
API --> QM
API --> Match
QM --> AI
AI --> LLM
Crawler --> PG
Match --> PG
QM --> Redis
API --> PG
AI --> Files
API --> Email
API --> SMS
Data Flow Architecture
sequenceDiagram
participant User
participant Frontend
participant API
participant Queue
participant AI
participant DB
participant Report
User->>Frontend: Upload CV & RFP
Frontend->>API: Submit Job Request
API->>API: Validate & Authenticate
API->>Queue: Add to Queue (Tier-based)
API-->>Frontend: Job ID & Status
Queue->>AI: Process Job
AI->>AI: Extract Text
AI->>AI: Call LLM Provider
AI->>DB: Store Analysis
AI->>Report: Generate PDF
Queue-->>API: Update Status
API-->>Frontend: Real-time Updates
Frontend-->>User: Show Progress
Report-->>User: Download Report
Deployment Architecture
graph TB
subgraph "Production Server"
subgraph "PM2 Cluster"
W1[Worker 1]
W2[Worker 2]
W3[Worker 3]
W4[Worker 4]
end
subgraph "Queue Workers"
T1[Tier 1 Premium]
T2[Tier 2 Business]
T3[Tier 3 Professional]
T4[Tier 4 Basic]
T5[Tier 5 Free]
end
subgraph "Services"
NGINX[NGINX Reverse Proxy]
PG[(PostgreSQL)]
REDIS[(Redis)]
end
end
subgraph "External"
CDN[CloudFlare CDN]
DNS[DNS]
end
DNS --> CDN
CDN --> NGINX
NGINX --> W1
NGINX --> W2
NGINX --> W3
NGINX --> W4
W1 --> PG
W2 --> PG
W3 --> PG
W4 --> PG
T1 --> REDIS
T2 --> REDIS
T3 --> REDIS
T4 --> REDIS
T5 --> REDIS
Database Schema
PostgreSQL Database: rfp_ai_matcher | User: rfpuser | Port: 5432
Core Tables Structure
erDiagram
COMPANIES ||--o{ SECTIONS : has
COMPANIES ||--o{ USERS : employs
SECTIONS ||--o{ TEAMS : contains
SECTIONS ||--o{ USERS : manages
TEAMS ||--o{ USERS : includes
USERS ||--o{ CONSULTANTS : manages
CONSULTANTS ||--o{ CVS : has
CONSULTANTS ||--o{ MATCHES : participates
JOBS ||--o{ MATCHES : requires
COMPANIES ||--o{ JOBS : posts
COMPANIES {
int company_id PK
string company_name
string org_number UK
string status
timestamp created_at
}
SECTIONS {
int section_id PK
int company_id FK
string section_name
int section_manager_id FK
timestamp created_at
}
TEAMS {
int team_id PK
int section_id FK
string team_name
timestamp created_at
}
USERS {
int user_id PK
int company_id FK
int section_id FK
int team_id FK
string email UK
string username
string password_hash
enum role
boolean is_active
timestamp last_login
}
CONSULTANTS {
uuid consultant_id PK
int user_id FK
string consultant_name
string email
string phone
enum availability_status
decimal hourly_rate
date assignment_end_date
}
CVS {
uuid cv_id PK
uuid consultant_id FK
string cv_name
string file_path
text parsed_content
jsonb metadata
timestamp uploaded_at
}
JOBS {
uuid job_id PK
int company_id FK
string title
text description
jsonb requirements
string provider
string external_id UK
timestamp crawled_at
}
MATCHES {
uuid match_id PK
uuid job_id FK
uuid consultant_id FK
uuid cv_id FK
decimal match_score
jsonb analysis_result
timestamp matched_at
}
Database Statistics
| Metric | Value | Description |
|---|---|---|
| Total Tables | 115 | All database tables including system tables |
| Total Indexes | 303 | Performance optimization indexes |
| Custom Types | 7 | ENUM types for roles, statuses, etc. |
| Storage Size | ~2GB | Current production database size |
| Daily Growth | ~50MB | Average daily data growth |
User Roles & Permissions
| Role | Access Level | Permissions |
|---|---|---|
| super_admin | 100 | Full system access, all companies |
| company_admin | 80 | Full company access, all sections |
| section_manager | 40 | Section-wide access, team management |
| consultant_manager | 30 | Consultant CV management |
| consultant | 10 | Own profile and CV access |
API Documentation
Base URL: https://sigmatalentsync.cloud |
Auth: JWT Bearer Token |
Content-Type: application/json
Authentication Endpoints
POST
/api/auth/login
{
"email": "[email protected]",
"password": "password123"
}
GET
/api/auth/me
Returns current authenticated user information
POST
/api/auth/logout
Invalidates current session token
POST
/api/auth/refresh
Refreshes JWT token before expiration
Queue System Endpoints
POST
/api/multi-queue/submit
{
"cvFile": [binary],
"rfpFile": [binary],
"tier": "premium",
"language": "sv",
"analysisType": "comprehensive"
}
GET
/api/multi-queue/stats
Returns real-time queue statistics for all tiers
GET
/api/multi-queue/status/{jobId}
Get status and progress of specific job
DELETE
/api/multi-queue/cancel/{jobId}
Cancel a pending job in the queue
Consultant Management
GET
/api/consultants
List all consultants with pagination and filters
POST
/api/consultants
Create new consultant profile
GET
/api/consultants/{consultant_id}
Get detailed consultant information
PUT
/api/consultants/{consultant_id}
Update consultant information
DELETE
/api/consultants/{consultant_id}
Delete consultant (soft delete)
Job Management
GET
/api/jobs
List all jobs with filters
GET
/api/jobs/{job_id}/matches
Get all consultant matches for a job
POST
/api/jobs/{job_id}/match
Trigger matching process for specific job
Multi-Tier Queue System
Membership Tiers
| Tier | Name | Priority | Threads | AI Model | Jobs/Hour | CheatAI |
|---|---|---|---|---|---|---|
| 1 | Premium | 10 | 40 (→55) | Gemini 2.5 Flash | 500 | ✅ |
| 2 | Business | 8 | 35 (→50) | Gemini 2.5 Pro | 400 | ✅ |
| 3 | Professional | 6 | 28 (→40) | X.AI Grok 3 Mini | 300 | ✅ |
| 4 | Basic | 4 | 20 (→35) | X.AI Grok 4 | 200 | ✅ |
| 5 | Free | 2 | 10 (→20) | Gemini 2.5 Flash Lite | 100 | ✅ |
Queue Processing Flow
graph LR
subgraph "Job Submission"
Upload[File Upload]
Validate[Validation]
Tier[Tier Assignment]
end
subgraph "Queue Management"
Queue1[Premium Queue]
Queue2[Business Queue]
Queue3[Professional Queue]
Queue4[Basic Queue]
Queue5[Free Queue]
end
subgraph "Processing"
Worker[Worker Pool]
AI[AI Processing]
Report[Report Generation]
end
subgraph "Delivery"
Store[Store Results]
Notify[Email Notification]
Download[Download Ready]
end
Upload --> Validate
Validate --> Tier
Tier --> Queue1
Tier --> Queue2
Tier --> Queue3
Tier --> Queue4
Tier --> Queue5
Queue1 --> Worker
Queue2 --> Worker
Queue3 --> Worker
Queue4 --> Worker
Queue5 --> Worker
Worker --> AI
AI --> Report
Report --> Store
Store --> Notify
Notify --> Download
Queue Features
⚡ Priority Processing
Higher tiers get priority in queue processing with dedicated resources
🔄 Auto-Scaling
Dynamic thread allocation based on queue load and tier priority
⚖️ Load Balancing
Automatic distribution of jobs across available workers
🔍 Real-time Monitoring
Live queue statistics and processing metrics
🎯 CheatAI Mode
Premium feature for AI-undetectable content generation
💰 Cost Optimization
Tier-based model selection for optimal cost/performance
Security Architecture
Authentication & Authorization
sequenceDiagram
participant Client
participant API
participant Auth
participant DB
participant Redis
Client->>API: Login Request
API->>DB: Validate Credentials
DB-->>API: User Data
API->>Auth: Generate JWT
Auth->>Redis: Store Session
Auth-->>API: JWT Token
API-->>Client: Token + User Info
Note over Client: Subsequent Requests
Client->>API: Request + JWT
API->>Auth: Validate Token
Auth->>Redis: Check Session
Redis-->>Auth: Session Valid
Auth-->>API: Authorized
API->>DB: Execute Request
DB-->>API: Response Data
API-->>Client: Response
Security Features
🔐 JWT Authentication
Secure token-based authentication with refresh tokens
HS256 algorithm, 24h expiry
🛡️ Password Security
bcrypt hashing with salt rounds
Salt rounds: 10
🚦 Rate Limiting
Tier-based request throttling
100 req/min (Free) - 1000 req/min (Premium)
🔒 HTTPS/TLS
SSL/TLS encryption for all traffic
TLS 1.3, HSTS enabled
🛑 Input Validation
SQL injection & XSS prevention
Parameterized queries, HTML sanitization
🌐 CORS Policy
Configured for production domain
Origin: sigmatalentsync.cloud
GDPR Compliance
| Requirement | Implementation | Status |
|---|---|---|
| Data Retention | Automated cleanup after 90 days of inactivity | Compliant |
| Audit Logging | Complete activity tracking with timestamps | Compliant |
| Consent Management | Explicit consent tracking for data processing | Compliant |
| Data Export | User data export API endpoint | Compliant |
| Right to Deletion | Soft delete with hard delete after 30 days | Compliant |
| Data Encryption | AES-256 for sensitive data at rest | Compliant |
User Journeys
Super Admin Journey
graph TD
Start[Login as Super Admin]
Dashboard[Access Admin Dashboard]
subgraph "Company Management"
CreateCompany[Create Company]
ManageCompanies[Manage Companies]
SetLimits[Set Resource Limits]
end
subgraph "User Management"
CreateAdmins[Create Company Admins]
ManageRoles[Manage User Roles]
AuditUsers[Audit User Activity]
end
subgraph "System Monitoring"
ViewMetrics[View System Metrics]
QueueStatus[Monitor Queue Status]
ErrorLogs[Review Error Logs]
end
subgraph "Configuration"
ConfigLLM[Configure LLM Providers]
SetPricing[Set Tier Pricing]
Features[Enable/Disable Features]
end
Start --> Dashboard
Dashboard --> CreateCompany
Dashboard --> ManageCompanies
Dashboard --> SetLimits
Dashboard --> CreateAdmins
Dashboard --> ManageRoles
Dashboard --> AuditUsers
Dashboard --> ViewMetrics
Dashboard --> QueueStatus
Dashboard --> ErrorLogs
Dashboard --> ConfigLLM
Dashboard --> SetPricing
Dashboard --> Features
Company Admin Journey
graph TD
Start[Login as Company Admin]
Dashboard[Company Dashboard]
subgraph "Organization Setup"
CreateSections[Create Sections]
CreateTeams[Create Teams]
AssignManagers[Assign Managers]
end
subgraph "User Management"
InviteUsers[Invite Users]
ManageUsers[Manage Users]
SetPermissions[Set Permissions]
end
subgraph "Consultant Management"
ViewConsultants[View All Consultants]
AssignConsultants[Assign to Teams]
ReviewCVs[Review CVs]
end
subgraph "Job Management"
CreateJobs[Create Job Postings]
ReviewMatches[Review Matches]
ApproveAssignments[Approve Assignments]
end
Start --> Dashboard
Dashboard --> CreateSections
CreateSections --> CreateTeams
CreateTeams --> AssignManagers
Dashboard --> InviteUsers
InviteUsers --> ManageUsers
ManageUsers --> SetPermissions
Dashboard --> ViewConsultants
ViewConsultants --> AssignConsultants
AssignConsultants --> ReviewCVs
Dashboard --> CreateJobs
CreateJobs --> ReviewMatches
ReviewMatches --> ApproveAssignments
Manager Journey
graph TD
Start[Login as Manager]
Dashboard[Team Dashboard]
subgraph "Team Management"
ViewTeam[View Team Members]
ManageConsultants[Manage Consultants]
ReviewPerformance[Review Performance]
end
subgraph "CV Management"
UploadCVs[Upload CVs]
UpdateCVs[Update CVs]
TagSkills[Tag Skills]
end
subgraph "Job Matching"
ViewJobs[View Available Jobs]
SubmitMatches[Submit for Matching]
ReviewResults[Review Match Results]
end
subgraph "Reporting"
TeamReports[Generate Team Reports]
UtilizationReports[Utilization Reports]
MatchReports[Match Success Reports]
end
Start --> Dashboard
Dashboard --> ViewTeam
ViewTeam --> ManageConsultants
ManageConsultants --> ReviewPerformance
Dashboard --> UploadCVs
UploadCVs --> UpdateCVs
UpdateCVs --> TagSkills
Dashboard --> ViewJobs
ViewJobs --> SubmitMatches
SubmitMatches --> ReviewResults
Dashboard --> TeamReports
TeamReports --> UtilizationReports
UtilizationReports --> MatchReports
Consultant Journey
graph TD
Start[Login as Consultant]
Profile[View Profile]
subgraph "Profile Management"
UpdateInfo[Update Personal Info]
UploadCV[Upload CV]
SetAvailability[Set Availability]
end
subgraph "Job Opportunities"
ViewMatches[View Job Matches]
ExpressInterest[Express Interest]
DeclineOffers[Decline Offers]
end
subgraph "Assignment"
CurrentAssignment[View Current Assignment]
UpdateStatus[Update Status]
LogHours[Log Hours]
end
subgraph "Communication"
Messages[View Messages]
ContactManager[Contact Manager]
Notifications[View Notifications]
end
Start --> Profile
Profile --> UpdateInfo
UpdateInfo --> UploadCV
UploadCV --> SetAvailability
Profile --> ViewMatches
ViewMatches --> ExpressInterest
ViewMatches --> DeclineOffers
Profile --> CurrentAssignment
CurrentAssignment --> UpdateStatus
UpdateStatus --> LogHours
Profile --> Messages
Messages --> ContactManager
Messages --> Notifications
Monitoring & Analytics
System Metrics
Queue Performance
98.5%
Success Rate
Average Processing
45s
Per Job
System Uptime
99.9%
Last 30 Days
Active Users
1,247
This Month
Monitoring Dashboard
graph TB
subgraph "Data Sources"
App[Application Logs]
DB[Database Metrics]
Queue[Queue Statistics]
Server[Server Metrics]
end
subgraph "Collection"
PM2[PM2 Metrics]
Custom[Custom Metrics]
Health[Health Checks]
end
subgraph "Processing"
Aggregate[Aggregation]
Alert[Alert Rules]
Store[Time Series DB]
end
subgraph "Visualization"
Dashboard[Admin Dashboard]
Reports[Reports]
Alerts[Alert Notifications]
end
App --> PM2
DB --> Custom
Queue --> Custom
Server --> Health
PM2 --> Aggregate
Custom --> Aggregate
Health --> Aggregate
Aggregate --> Store
Aggregate --> Alert
Store --> Dashboard
Store --> Reports
Alert --> Alerts
Logging Architecture
| Log Type | Location | Retention | Purpose |
|---|---|---|---|
| Error Logs | /logs/error.log | 90 days | Application errors and exceptions |
| Access Logs | /logs/access.log | 30 days | HTTP request/response logging |
| Queue Logs | /logs/queue.log | 7 days | Queue processing events |
| Audit Logs | PostgreSQL audit_logs table | 1 year | User actions and data changes |
| Debug Logs | /logs/debug.log | 24 hours | Detailed debugging information |
Key Performance Indicators (KPIs)
📊 Job Processing Rate
Average: 500 jobs/hour
Peak: 1,200 jobs/hour
⏱️ Response Time
API: <200ms (p95)
Database: <50ms (p95)
💾 Storage Utilization
Database: 2GB / 10GB
Files: 50GB / 100GB
🔄 Queue Efficiency
Throughput: 85%
Idle time: <15%
🎯 Match Accuracy
Precision: 92%
Recall: 88%
👥 User Engagement
DAU: 450 users
MAU: 1,247 users