Queensway Workflow

The Queensway workflow uses our Guestline chain tasks to build a complete, end-to-end ETL pipeline for Queensway properties.

Workflow Overview

This workflow follows a standard ETL pattern with four main stages:

1. Raw Data Ingestion & Crawling - Using GuestlineRawTask to ingest and parse roompick and personprofile payloads
2. Data Cleaning - Using CleanGuestlineTask to standardize the payloads into Clean* models
3. Processing - Running the standard ProcessingTask subtasks (with optional skip/insert for chain-specific merge logic)
4. Report Generation - Producing reports with ReportsTask and writing to BigQuery

Pipeline Architecture

┌─────────────────────┐
│  GuestlineRawTask   │  ← Ingestion + Crawling
└──────────┬──────────┘
           │ RawRoompicksModel
           │ RawPersonprofilesModel
           ↓
┌─────────────────────┐
│ CleanGuestlineTask  │  ← Data Cleaning
└──────────┬──────────┘
           │ CleanGuestModel
           │ CleanReservationModel
           │ CleanRoomModel
           ↓
┌─────────────────────┐
│   ProcessingTask    │  ← Business Logic
└──────────┬──────────┘
           │ ProcessedGuestModel
           │ ProcessedReservationModel
           │ ProcessedRoomModel
           ↓
┌─────────────────────┐
│    ReportsTask      │  ← Analytics Reports
└─────────────────────┘

Configuration

• GuestlineRawTask(job_context=..., skip_ingestion=False, sync_dates=...) - controls ingest vs use of existing files
• model_configs may override default raw database names:

{
  "RawRoompicksModel": { "database": "etl_gp_raw_queensway" },
  "RawPersonprofilesModel": { "database": "etl_gp_raw_queensway" }
}

Execution Flow

When orchestrator.run() is called for Queensway (see jobs/queensway_task.py):

1. GuestlineRawTask writes ingest/raw files for roompicks and personprofiles.
2. CleanGuestlineTask standardizes fields and produces cleaned models.
3. ProcessingTask transforms cleaned models into processed tables; optionally skip GuestMatchingTask as Queensway does in certain runs.
4. ReportsTask writes final analytics tables into BigQuery.

Additional Notes

• Queensway uses the Guestline chain tasks, so any improvements in guestline chain tasks will benefit Queensway.
• The CleanGuestlineTask contains Queensway-specific helpers and mapping logic (address fields, company/travel agent mapping, exclude-from-mailings flags).
• Keep JobContext configured with the correct chain_id (queensway) and partion count for performance on batch processing.

Recommended Jobs

• jobs/queensway_task.py — Example job that demonstrates a full run of the Queensway pipeline (ingest, clean, processing, reports) and shows usage of skip_subtasks and skip_task.