> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tracklysms.com/llms.txt
> Use this file to discover all available pages before exploring further.
# ML Creative Selection
> Automatically select the best creative for each contact
Trackly SMS uses machine learning to select the optimal creative for each contact, maximizing clicks and revenue.
## How It Works
Instead of manually choosing which message to send, the ML system:
1. **Analyzes contact history** (clicks, revenue, engagement patterns)
2. **Predicts performance** for each available creative
3. **Selects the winner** for that specific contact
4. **Learns from results** to improve over time
## Enabling Automated Selection
### Campaign Setup
1. Create a campaign in **Automated Mode**
2. Configure the automated percentage:
```
Automated selection: 80%
Manual fallback: 20%
```
* 80% of contacts get ML-selected creatives
* 20% get the default creative (control group)
### Available Creatives
The ML system chooses from creatives marked as **Automated** type:
1. Go to **Creatives**
2. Mark creatives as "Automated" type
3. These become candidates for ML selection
## The Selection Model
The production system uses **Thompson Sampling** with Beta distribution posteriors as the primary selection algorithm. (LightGBM exists as a placeholder that falls back to random selection.)
### How Thompson Sampling Works
Thompson Sampling is a Bayesian algorithm that balances **exploration** (trying less-tested creatives) with **exploitation** (favoring proven winners). For each send:
1. Sample a score from each creative's Beta distribution posterior
2. Select the creative with the highest sampled score
3. After delivery, update the posterior with observed results
### Two Selection Modes
| Mode | Description |
| ---------------------- | -------------------------------------------------------------------- |
| `thompson_single_step` | Scores all creatives together by sampled Revenue Per Send (RPS) |
| `thompson_two_step` | Picks the best offer first, then the best creative within that offer |
### Metric Optimized
The algorithm optimizes **Revenue Per Send (RPS)**:
```
RPS = CTR × Conversion Rate × Offer Value
```
For **CTO (cost-to-operator) offers**, conversion rate is omitted:
```
RPS = CTR × Offer Value
```
Offers with payouts below $20 are floored to $120 for scoring purposes, preventing bias against low-payout but high-volume offers.
### Pre-Computed Parameters
Beta distribution alpha/beta parameters are computed by a **daily job** (`thompson_stats_job`) and stored in `ThompsonSamplingParams` per account. This avoids database queries during the send hot path.
### Features Used
The model considers:
| Feature | Description |
| ------------------ | ----------------------------------------------------------- |
| Send count | How many messages they've received |
| Click count | Historical click behavior |
| Last click recency | Days since last click |
| Revenue history | Past purchase behavior |
| Time of day | When they typically engage |
| Creative history | Which creatives they've clicked before |
| Custom fields | Contact-level custom field data used as additional features |
### Model Training
Models are trained on your account's historical data:
1. **Click model**: Predicts probability of clicking
2. **Revenue model**: Predicts expected revenue
Training runs automatically with a **configurable frequency** set in your account's automated selection settings. The system incorporates recent performance data on each training run.
### Model Metrics
The system tracks detailed model performance metrics including:
* **Log loss** — measures prediction calibration
* **MAE (Mean Absolute Error)** — measures revenue prediction accuracy
* **Feature importance** — ranks which features drive predictions most
### Predictions
For each contact, the model predicts:
```
Creative A: 12% click probability, $0.45 expected revenue
Creative B: 8% click probability, $0.52 expected revenue
Creative C: 15% click probability, $0.38 expected revenue
```
Selection can optimize for clicks OR revenue.
## Configuration
Optimization settings are configured at the **account level** in `automated_selection_settings`. Campaigns only control `automated_percent` — the percentage of sends that use automated selection vs. the manual fallback creative.
### Optimization Goal
Choose what to optimize for:
| Goal | Best For |
| ----------- | ------------------------ |
| **Clicks** | Engagement, list warming |
| **Revenue** | Direct response, sales |
### Cooldown Settings
Cooldown is enforced **per-contact, per-creative** and is configured in your account's `automated_selection_settings`. When a contact receives a specific creative, that creative becomes ineligible for that contact until the cooldown period expires.
```
Example: If cooldown is set to 5 days and a contact received Creative A today,
Creative A won't be selected for that contact for 5 days.
Other creatives remain eligible.
```
### Selection Methods
The system supports several selection methods, configured at the account level:
| Method | Description |
| -------------------------- | ------------------------------------------------------------------------------- |
| **`thompson_single_step`** | Scores all creatives together by sampled RPS — the primary production algorithm |
| **`thompson_two_step`** | Picks the best offer first, then the best creative within that offer |
| **`lightgbm`** | Placeholder — currently falls back to random selection |
| **`random`** | Randomly selects from eligible creatives (useful for gathering baseline data) |
| **`manual`** | Uses a filtered subset with random or round-robin rotation |
Multiple methods can be configured simultaneously with `traffic_pct` weights, routing different percentages of traffic to different algorithms.
Thompson sampling naturally balances exploration and exploitation: creatives with uncertain performance get tested more frequently, while proven performers are selected more often. This replaces the need for a fixed exploration rate.
## Performance Tracking
### Model Metrics
View model performance in **Reports > ML Metrics**:
| Metric | Description | Good Value |
| ---------------------- | ------------------------------ | --------------- |
| **Log Loss** | Prediction calibration quality | Lower is better |
| **MAE** | Revenue prediction accuracy | Lower is better |
| **Feature importance** | What drives predictions | - |
### A/B vs Automated
Compare automated selection against manual:
```
Automated (80%): 14% click rate, $0.52 revenue/send
Control (20%): 11% click rate, $0.41 revenue/send
Lift: +27% clicks, +27% revenue
```
## When to Use ML Selection
* Large contact list (5,000+)
* Multiple creatives to choose from (3+)
* Enough historical data (10,000+ sends)
* Measurable conversion goals
* Small list (\< 1,000 contacts)
* Single creative
* New account with limited history
* Highly targeted sends (already segmented)
## Cold Start / Learning Pool
Creatives with fewer than **3,000 raw sends** are placed in a **learning pool**. A configurable percentage of traffic (default **10%**) is allocated to learning creatives, which are selected randomly to gather baseline data. The remaining 90% uses Thompson Sampling scoring.
Once a creative crosses the 3,000-send threshold, it exits the learning pool and competes on its sampled RPS score.
### Graduation Warmup Ramp
The learning percentage you configure applies only once your account holds a real competitive field. Until **20 creatives** have graduated past the min sends threshold, the platform automatically raises the effective learning share:
```
effective learning share = max(configured learning %, 1 − graduates / 20)
```
| Graduated creatives | Learning share | Thompson (exploit) share |
| ------------------- | -------------- | ------------------------ |
| 0–1 | 100% | 0% (exploit gate) |
| 2 | 90% | 10% |
| 5 | 75% | 25% |
| 10 | 50% | 50% |
| 20+ | your setting | the rest |
Without this ramp, the first creative or two to graduate would immediately capture \~90% of all sends — your contacts would receive the same one or two messages repeatedly while hundreds of other creatives starved. Each additional graduate shifts 5% of traffic back to Thompson exploitation, so the transition from warmup to full optimization is gradual and self-accelerating. Accounts with 20+ graduated creatives are unaffected — the configured learning percentage applies exactly.
The ramp also applies when learning percentage is set to 0: with a small graduated field, exploration continues regardless, because without it no additional creative could ever graduate.
### New Accounts
Without historical data:
1. All creatives start in the learning pool
2. Traffic is split randomly until enough data accumulates
3. After 3,000+ sends per creative, Thompson Sampling takes over
4. The daily `thompson_stats_job` computes Beta distribution parameters
### New Creatives
New creatives automatically enter the learning pool:
* They receive guaranteed exposure from the 10% learning allocation
* Performance data accumulates over \~3,000 sends
* Strong performers graduate to Thompson scoring and rise to the top
## Monitoring
### Dashboard Indicators
Watch for:
* **Model health**: Is the model performing well?
* **Creative diversity**: Are all creatives getting selected?
* **Performance trends**: Is click rate improving?
### Alerts
Set up alerts for:
* Model degradation
* Single creative dominating (may indicate overfitting)
* Performance drops
## Best Practices
Keep 5-10 active automated creatives. Too few limits ML effectiveness.
Add new creatives periodically. Stale content loses effectiveness.
Always maintain some manual sends to measure ML lift.
Understanding what drives predictions helps create better creatives.
## Next Steps
Set up automated campaigns
Analyze ML performance