chore: establish secure initial project baseline
This commit is contained in:
MrARM
@@ -0,0 +1,4 @@
|
||||
*
|
||||
!requirements.txt
|
||||
!app/
|
||||
!app/collector.py
|
||||
@@ -0,0 +1,13 @@
|
||||
# API account: use an account you own or are authorized to monitor.
|
||||
API_BASE_URL=https://iop-dev.strndev.com
|
||||
API_USERNAME=replace-me
|
||||
API_PASSWORD=replace-me
|
||||
POLL_SECONDS=300
|
||||
|
||||
# Use distinct, long random values before exposing any service beyond localhost.
|
||||
POSTGRES_DB=play_history
|
||||
POSTGRES_USER=play_history
|
||||
POSTGRES_PASSWORD=replace-with-a-long-random-password
|
||||
GRAFANA_ADMIN_USER=admin
|
||||
GRAFANA_ADMIN_PASSWORD=replace-with-a-long-random-password
|
||||
GRAFANA_PORT=3000
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Secrets and machine-specific configuration
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
|
||||
# Local runtime, tooling, and agent artifacts
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.ruff_cache/
|
||||
.pi-subagents/
|
||||
|
||||
# Collected account activity may contain private data
|
||||
play-history.jsonl
|
||||
@@ -0,0 +1,63 @@
|
||||
# Stern Insider API reconstruction
|
||||
|
||||
**Reason for existence:** provides a reproducible, OpenAPI-compatible inventory of the public API surface at `https://iop-dev.strndev.com/` without publishing opaque webhook secrets.
|
||||
|
||||
## Deliverables
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `openapi.reconstructed.json` | Valid OpenAPI 3.0.3 manifest; **257** URLconf paths and **249** observed operations. |
|
||||
| `endpoint-options-evidence.json` | Sanitized, read-only `OPTIONS` evidence for each path. |
|
||||
| `endpoint-get-evidence.json` | Sanitized GET status/content-type/byte-count and inferred JSON shapes; no response values. |
|
||||
| `endpoint-validation-evidence.json` | Sanitized models from intentionally invalid credential/registration validation requests. |
|
||||
| `django-404-urlpatterns.routes.txt` | Sanitized path inventory parsed from Django's public DEBUG 404 page. |
|
||||
| `reconstruct_openapi.py` | Re-runs the read-only discovery process. |
|
||||
|
||||
## Evidence and confidence
|
||||
|
||||
The manifest was generated from Django's public DEBUG 404 URLconf, then `OPTIONS` was issued with `Accept: application/json` for every discovered API/webhook route. Public GET routes were fetched to model response fields across all returned array items. Six credential/registration validation routes were sent `{}`; their observed JSON validation models are included without storing response values. This exposed methods, authentication challenges, five request serializers, 20 public GET response shapes, and six validation-error response models.
|
||||
|
||||
| Item | Result |
|
||||
| --- | ---: |
|
||||
| Discovered API/webhook paths | 257 |
|
||||
| Observed methods | 123 GET, 122 POST, 1 PATCH, 3 DELETE |
|
||||
| `OPTIONS` returned 200 | 51 |
|
||||
| Auth-gated `OPTIONS` (401/403) | 187 |
|
||||
| Public GET response shapes inferred | 20 |
|
||||
| Request-body serializers exposed | 5 |
|
||||
| Operations with concrete request examples | 5 |
|
||||
| Operations with concrete response examples | 26 |
|
||||
| URLconf paths with no inferable method | 8 |
|
||||
|
||||
The request schemas directly exposed by Django REST Framework are present for:
|
||||
|
||||
- `POST /api/v2/token/` — `username`, `password`
|
||||
- `POST /api/v2/token/refresh/` and `/api/v3/token/refresh/` — `refresh`, `access`
|
||||
- `POST /api/v4/auth/login/` — `request_metadata`, `email`, `password`, `remember_me`
|
||||
- `POST /api/v4/auth/register/` — `first_name`, `last_name`, `username`, `initials`, `email_consent`, `background_color_id`, `avatar_id`, `location_info`, `age_restricted`
|
||||
|
||||
## Authentication
|
||||
|
||||
The manifest defines `bearerAuth` as JWT bearer authentication. Server evidence includes `WWW-Authenticate: Bearer realm="api"` on protected v4 endpoints. The URLconf also exposes legacy/session authentication and token routes, including `/api-token-auth/`, `/api/v2/token/`, `/api/v2/token/refresh/`, and logout endpoints. No account was created: registration or login is unnecessary for this read-only reconstruction.
|
||||
|
||||
## Important limitations
|
||||
|
||||
- This is a reconstruction, not an official contract. A `401`/`403` response can hide serializer metadata and response schemas.
|
||||
- GET schemas are sampled from one public response only; optionality, enum domains, pagination, and item variation are not guaranteed.
|
||||
- Eight URLconf paths are retained with `x-reconstruction-status` but have no operation because `OPTIONS` supplied no `Allow` header (parameterized detail/download routes, SSO, specific webhooks, and several server-error routes).
|
||||
- Opaque webhook tokens exposed by DEBUG are replaced with `{webhook_secret}`. Do not recover or commit those tokens.
|
||||
- Re-running the script causes one deliberate 404, `OPTIONS`, public GET requests, and six deliberately invalid POST validation requests. It performs no login, registration, successful POST, PATCH, or DELETE.
|
||||
|
||||
## HTML routes excluded from the manifest
|
||||
|
||||
These are server-rendered/admin/tooling routes rather than the JSON API explorer surface: `/`, `/admin/`, `/dashboard/`, `/auth/`, `/accounts/`, `/tools/`, `/business_registration_invitation/`, `/index`, `/login/`, `/pro/`, `/machine_registration/`, `/game_alerts/`, `/game_audits/`, `/game_play/`, `/clear_game_play/`, `/generate_emails/`, `/resend_confirmation_email/`, `/generate_notifications/`, `/tools/create_user/`, `/password-change/`, `/password-change/done/`, `/favicon.ico`, `/api-auth/`, and `/tz_detect/`.
|
||||
|
||||
Two opaque-named diagnostic HTML routes are intentionally not reproduced. `healthz/` is also excluded because it is an operational health endpoint, not a Django API explorer endpoint.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
cd /Users/jordan/Downloads/stern-api
|
||||
python3 reconstruct_openapi.py
|
||||
npx --yes @apidevtools/swagger-cli@latest validate openapi.reconstructed.json
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY app/ ./app/
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
CMD ["python", "-m", "app.collector"]
|
||||
@@ -0,0 +1,39 @@
|
||||
# Play-history Grafana stack
|
||||
|
||||
## What it does
|
||||
|
||||
`docker compose` runs three services:
|
||||
|
||||
- **collector** logs in as the configured account, polls `/api/v1/portal/my_activity/`, follows same-origin pagination, and stores only new activity states.
|
||||
- **PostgreSQL** stores normalized activity records and their original API payload.
|
||||
- **Grafana** provisions a PostgreSQL datasource plus a dashboard for maximum score, total plays, and the latest activity table.
|
||||
|
||||
The collector uses the authenticated activity endpoint that returned 28 records in `play-history.jsonl`. It does not scrape unrelated accounts.
|
||||
|
||||
## Start
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env: set API_USERNAME, API_PASSWORD, POSTGRES_PASSWORD, and GRAFANA_ADMIN_PASSWORD.
|
||||
docker compose up --build -d
|
||||
```
|
||||
|
||||
Open `http://localhost:3000`, log in with `GRAFANA_ADMIN_USER` and `GRAFANA_ADMIN_PASSWORD`, then open **Dashboards → Play History → Stern Play History**.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
docker compose logs collector --tail=50
|
||||
docker compose exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
|
||||
-c 'SELECT count(*) AS score_states FROM activity_snapshot;'
|
||||
```
|
||||
|
||||
The first poll should report `collected 28 activities` and insert the current states. Later polls only insert new or changed score states, so the Grafana time series does not duplicate unchanged data.
|
||||
|
||||
## Operations
|
||||
|
||||
- Change `POLL_SECONDS` in `.env`; the collector enforces a 30-second minimum.
|
||||
- Grafana is bound to localhost by default. Do not expose it publicly without TLS and a stronger access-control layer.
|
||||
- `API_PASSWORD`, JWTs, and refresh tokens are never stored in PostgreSQL or the collector logs.
|
||||
- Use `docker compose down` to stop services. Add `-v` only if you intentionally want to delete stored history.
|
||||
@@ -0,0 +1,203 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Poll a Stern Insider account's activity endpoint and persist score changes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
from urllib.error import HTTPError, URLError
|
||||
from urllib.parse import urljoin, urlparse
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
psycopg = importlib.import_module("psycopg")
|
||||
|
||||
LOG = logging.getLogger("stern_collector")
|
||||
USER_AGENT = "stern-play-history-collector/1.0"
|
||||
|
||||
|
||||
def setting(name: str) -> str:
|
||||
value = os.environ.get(name)
|
||||
if not value:
|
||||
raise RuntimeError(f"{name} must be set")
|
||||
return value
|
||||
|
||||
|
||||
def request_json(
|
||||
url: str,
|
||||
method: str = "GET",
|
||||
payload: dict[str, str] | None = None,
|
||||
token: str | None = None,
|
||||
) -> tuple[int, Any]:
|
||||
body = json.dumps(payload).encode() if payload is not None else None
|
||||
headers = {"Accept": "application/json", "User-Agent": USER_AGENT}
|
||||
if body:
|
||||
headers["Content-Type"] = "application/json"
|
||||
if token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
request = Request(url, data=body, headers=headers, method=method)
|
||||
try:
|
||||
with urlopen(request, timeout=30) as response:
|
||||
status, raw = response.status, response.read()
|
||||
except HTTPError as error:
|
||||
status, raw = error.code, error.read()
|
||||
except URLError as error:
|
||||
raise RuntimeError(f"network error: {error.reason}") from error
|
||||
try:
|
||||
return status, json.loads(raw)
|
||||
except (UnicodeDecodeError, json.JSONDecodeError):
|
||||
return status, {"non_json_response_bytes": len(raw)}
|
||||
|
||||
|
||||
def authenticate(base_url: str) -> str:
|
||||
credential_fields = ("username", "password")
|
||||
credential_values = (setting("API_USERNAME"), setting("API_PASSWORD"))
|
||||
credentials: dict[str, str] = dict(
|
||||
zip(credential_fields, credential_values, strict=True)
|
||||
)
|
||||
status, response = request_json(
|
||||
urljoin(base_url, "api/v2/token/"),
|
||||
method="POST",
|
||||
payload=credentials,
|
||||
)
|
||||
if (
|
||||
status != 200
|
||||
or not isinstance(response, dict)
|
||||
or not isinstance(response.get("access"), str)
|
||||
):
|
||||
raise RuntimeError(f"authentication failed (HTTP {status})")
|
||||
return response["access"]
|
||||
|
||||
|
||||
def activity_pages(base_url: str, token: str) -> list[dict[str, Any]]:
|
||||
origin = urlparse(base_url).netloc
|
||||
url = urljoin(base_url, "api/v1/portal/my_activity/")
|
||||
entries: list[dict[str, Any]] = []
|
||||
while url:
|
||||
if urlparse(url).netloc != origin:
|
||||
raise RuntimeError("API pagination pointed to a different origin")
|
||||
status, payload = request_json(url, token=token)
|
||||
if status != 200 or not isinstance(payload, dict):
|
||||
raise RuntimeError(f"activity request failed (HTTP {status})")
|
||||
page = payload.get("results")
|
||||
if not isinstance(page, list):
|
||||
raise RuntimeError("activity response did not contain a results list")
|
||||
entries.extend(item for item in page if isinstance(item, dict))
|
||||
next_page = payload.get("next")
|
||||
url = (
|
||||
urljoin(base_url, next_page)
|
||||
if isinstance(next_page, str) and next_page
|
||||
else ""
|
||||
)
|
||||
return entries
|
||||
|
||||
|
||||
def database_connection() -> Any:
|
||||
return psycopg.connect(
|
||||
host=setting("POSTGRES_HOST"),
|
||||
port=os.environ.get("POSTGRES_PORT", "5432"),
|
||||
dbname=setting("POSTGRES_DB"),
|
||||
user=setting("POSTGRES_USER"),
|
||||
password=setting("POSTGRES_PASSWORD"),
|
||||
)
|
||||
|
||||
|
||||
def required_text(entry: dict[str, Any], field: str) -> str:
|
||||
value = entry.get(field)
|
||||
if value is None:
|
||||
raise ValueError(f"activity item omitted {field}")
|
||||
return str(value)
|
||||
|
||||
|
||||
def optional_int(entry: dict[str, Any], field: str) -> int | None:
|
||||
value = entry.get(field)
|
||||
if value in (None, ""):
|
||||
return None
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError) as error:
|
||||
raise ValueError(f"activity item has a non-integer {field}") from error
|
||||
|
||||
|
||||
def polling_interval() -> int:
|
||||
try:
|
||||
return max(30, int(os.environ.get("POLL_SECONDS", "300")))
|
||||
except ValueError as error:
|
||||
raise RuntimeError("POLL_SECONDS must be an integer") from error
|
||||
|
||||
|
||||
def persist(entries: list[dict[str, Any]]) -> int:
|
||||
observed_at = datetime.now(timezone.utc)
|
||||
inserted = 0
|
||||
with database_connection() as connection, connection.cursor() as cursor:
|
||||
for entry in entries:
|
||||
canonical = json.dumps(entry, sort_keys=True, separators=(",", ":"))
|
||||
source_hash = hashlib.sha256(canonical.encode()).hexdigest()
|
||||
try:
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO activity_snapshot (
|
||||
source_hash, observed_at, activity_date, game_model_id,
|
||||
game_model_type, location_id, location_name, max_score,
|
||||
total_plays, raw_activity
|
||||
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s::jsonb)
|
||||
ON CONFLICT (source_hash) DO NOTHING
|
||||
""",
|
||||
(
|
||||
source_hash,
|
||||
observed_at,
|
||||
required_text(entry, "date"),
|
||||
required_text(entry, "game_model_id"),
|
||||
required_text(entry, "game_model_type"),
|
||||
required_text(entry, "location_id"),
|
||||
required_text(entry, "location_name"),
|
||||
optional_int(entry, "max_score"),
|
||||
optional_int(entry, "total_plays"),
|
||||
canonical,
|
||||
),
|
||||
)
|
||||
inserted += cursor.rowcount
|
||||
except (ValueError, psycopg.Error) as error:
|
||||
LOG.warning("skipping malformed activity item: %s", error)
|
||||
cursor.execute(
|
||||
"INSERT INTO collection_run (observed_at, item_count, inserted_count) VALUES (%s, %s, %s)",
|
||||
(observed_at, len(entries), inserted),
|
||||
)
|
||||
return inserted
|
||||
|
||||
|
||||
def run_once(base_url: str) -> None:
|
||||
token = authenticate(base_url)
|
||||
entries = activity_pages(base_url, token)
|
||||
inserted = persist(entries)
|
||||
LOG.info(
|
||||
"collected %d activities; inserted %d new score states", len(entries), inserted
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
logging.basicConfig(
|
||||
level=os.environ.get("LOG_LEVEL", "INFO"),
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
)
|
||||
base_url = setting("API_BASE_URL").rstrip("/") + "/"
|
||||
if urlparse(base_url).scheme != "https" and os.environ.get("ALLOW_HTTP") != "true":
|
||||
raise RuntimeError(
|
||||
"API_BASE_URL must use HTTPS; set ALLOW_HTTP=true only for a trusted local server"
|
||||
)
|
||||
interval = polling_interval()
|
||||
while True:
|
||||
try:
|
||||
run_once(base_url)
|
||||
except Exception:
|
||||
LOG.exception("collection cycle failed")
|
||||
time.sleep(interval)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,191 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Collect authenticated Stern Insider play-history snapshots as JSON Lines.
|
||||
|
||||
Uses the legacy JWT token endpoint by default. It never prints or writes the
|
||||
password, access token, or refresh token.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import getpass
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.error import HTTPError, URLError
|
||||
from urllib.parse import urlencode, urljoin, urlparse
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
DEFAULT_ENDPOINTS = (
|
||||
"/api/v1/portal/my_activity/",
|
||||
"/api/v1/portal/game_machine_high_scores/",
|
||||
"/api/v1/portal/user_title_stats/",
|
||||
"/api/v1/portal/user_stats/",
|
||||
)
|
||||
USER_AGENT = "stern-play-history-collector/1.0"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--base-url", required=True, help="API origin, e.g. https://iop-dev.strndev.com"
|
||||
)
|
||||
parser.add_argument("--username", help="Account username; prompts if omitted")
|
||||
parser.add_argument(
|
||||
"--password", help="Account password; prompts securely if omitted"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--token-path",
|
||||
default="/api/v2/token/",
|
||||
help="JWT login path (default: /api/v2/token/)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--endpoint",
|
||||
action="append",
|
||||
help="Authenticated GET endpoint to collect; repeat to override the defaults",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--query",
|
||||
action="append",
|
||||
default=[],
|
||||
metavar="KEY=VALUE",
|
||||
help="Query parameter applied to every endpoint; repeatable",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=Path,
|
||||
default=Path("play-history.jsonl"),
|
||||
help="JSON Lines snapshot file",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--interval",
|
||||
type=float,
|
||||
default=0,
|
||||
help="Seconds between collection cycles; 0 runs once",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow-http",
|
||||
action="store_true",
|
||||
help="Permit an insecure HTTP API URL (not recommended)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def normalized_base_url(raw_url: str, allow_http: bool) -> str:
|
||||
parsed = urlparse(raw_url)
|
||||
if parsed.scheme not in {"https", "http"} or not parsed.netloc:
|
||||
raise ValueError("--base-url must be an absolute http(s) URL")
|
||||
if parsed.scheme != "https" and not allow_http:
|
||||
raise ValueError("refusing HTTP; use HTTPS or pass --allow-http explicitly")
|
||||
return raw_url.rstrip("/") + "/"
|
||||
|
||||
|
||||
def parse_query(values: list[str]) -> dict[str, str]:
|
||||
query: dict[str, str] = {}
|
||||
for value in values:
|
||||
key, separator, item = value.partition("=")
|
||||
if not separator or not key:
|
||||
raise ValueError(f"invalid --query {value!r}; use KEY=VALUE")
|
||||
query[key] = item
|
||||
return query
|
||||
|
||||
|
||||
def request_json(
|
||||
url: str,
|
||||
method: str = "GET",
|
||||
payload: dict[str, Any] | None = None,
|
||||
access_token: str | None = None,
|
||||
) -> tuple[int, Any]:
|
||||
body = json.dumps(payload).encode() if payload is not None else None
|
||||
headers = {"Accept": "application/json", "User-Agent": USER_AGENT}
|
||||
if body is not None:
|
||||
headers["Content-Type"] = "application/json"
|
||||
if access_token:
|
||||
headers["Authorization"] = f"Bearer {access_token}"
|
||||
request = Request(url, data=body, headers=headers, method=method)
|
||||
try:
|
||||
with urlopen(request, timeout=30) as response:
|
||||
status, raw = response.status, response.read()
|
||||
except HTTPError as error:
|
||||
status, raw = error.code, error.read()
|
||||
except URLError as error:
|
||||
raise RuntimeError(f"network error for {url}: {error.reason}") from error
|
||||
|
||||
try:
|
||||
return status, json.loads(raw)
|
||||
except (UnicodeDecodeError, json.JSONDecodeError):
|
||||
return status, {"non_json_response_bytes": len(raw)}
|
||||
|
||||
|
||||
def authenticate(base_url: str, token_path: str, username: str, password: str) -> str:
|
||||
status, response = request_json(
|
||||
urljoin(base_url, token_path.lstrip("/")),
|
||||
method="POST",
|
||||
payload={"username": username, "password": password},
|
||||
)
|
||||
if (
|
||||
status != 200
|
||||
or not isinstance(response, dict)
|
||||
or not isinstance(response.get("access"), str)
|
||||
):
|
||||
raise RuntimeError(f"login failed with HTTP {status}: {json.dumps(response)}")
|
||||
return response["access"]
|
||||
|
||||
|
||||
def collect_once(
|
||||
base_url: str,
|
||||
endpoints: tuple[str, ...],
|
||||
query: dict[str, str],
|
||||
access_token: str,
|
||||
output: Path,
|
||||
) -> None:
|
||||
retrieved_at = datetime.now(timezone.utc).isoformat()
|
||||
with output.open("a", encoding="utf-8") as destination:
|
||||
for endpoint in endpoints:
|
||||
url = urljoin(base_url, endpoint.lstrip("/"))
|
||||
if query:
|
||||
url += "?" + urlencode(query)
|
||||
status, data = request_json(url, access_token=access_token)
|
||||
snapshot = {
|
||||
"retrieved_at": retrieved_at,
|
||||
"endpoint": endpoint,
|
||||
"status": status,
|
||||
"data": data,
|
||||
}
|
||||
destination.write(
|
||||
json.dumps(snapshot, separators=(",", ":"), default=str) + "\n"
|
||||
)
|
||||
print(f"{endpoint}: HTTP {status}", file=sys.stderr)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
try:
|
||||
base_url = normalized_base_url(args.base_url, args.allow_http)
|
||||
query = parse_query(args.query)
|
||||
except ValueError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
username = args.username or input("Username: ")
|
||||
password = args.password or getpass.getpass("Password: ")
|
||||
endpoints = tuple(args.endpoint) if args.endpoint else DEFAULT_ENDPOINTS
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
try:
|
||||
access_token = authenticate(base_url, args.token_path, username, password)
|
||||
while True:
|
||||
collect_once(base_url, endpoints, query, access_token, args.output)
|
||||
if args.interval <= 0:
|
||||
return 0
|
||||
time.sleep(args.interval)
|
||||
except RuntimeError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
CREATE TABLE IF NOT EXISTS activity_snapshot (
|
||||
source_hash TEXT PRIMARY KEY,
|
||||
observed_at TIMESTAMPTZ NOT NULL,
|
||||
activity_date DATE NOT NULL,
|
||||
game_model_id TEXT NOT NULL,
|
||||
game_model_type TEXT NOT NULL,
|
||||
location_id TEXT NOT NULL,
|
||||
location_name TEXT NOT NULL,
|
||||
max_score BIGINT,
|
||||
total_plays INTEGER,
|
||||
raw_activity JSONB NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS activity_snapshot_observed_at_idx
|
||||
ON activity_snapshot (observed_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS activity_snapshot_model_idx
|
||||
ON activity_snapshot (game_model_type, game_model_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS collection_run (
|
||||
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
observed_at TIMESTAMPTZ NOT NULL,
|
||||
item_count INTEGER NOT NULL,
|
||||
inserted_count INTEGER NOT NULL
|
||||
);
|
||||
@@ -0,0 +1,257 @@
|
||||
/api/v1/canadian_province_codes/
|
||||
/api/v1/configs/
|
||||
/api/v1/convert_utc_to_tz_time/
|
||||
/api/v1/country_codes/
|
||||
/api/v1/echo/
|
||||
/api/v1/forums/categories/
|
||||
/api/v1/forums/categories_with_topics/
|
||||
/api/v1/game/alert_events/
|
||||
/api/v1/game/change_home_team/
|
||||
/api/v1/game/change_home_team_group/
|
||||
/api/v1/game/change_home_team_individual/
|
||||
/api/v1/game/combined_shared_properties/
|
||||
/api/v1/game/game_configuration/
|
||||
/api/v1/game/heartbeat/
|
||||
/api/v1/game/high_score_events/
|
||||
/api/v1/game/machine_audits/
|
||||
/api/v1/game/payment_events/
|
||||
/api/v1/game/player_list_info/
|
||||
/api/v1/game/state/
|
||||
/api/v1/game/sync_message_number/
|
||||
/api/v1/leaderboard_restricted_country_codes/
|
||||
/api/v1/location_types/
|
||||
/api/v1/log_message/
|
||||
/api/v1/minigames/
|
||||
/api/v1/notification_types/
|
||||
/api/v1/ping/
|
||||
/api/v1/portal/
|
||||
/api/v1/portal/adcards/
|
||||
/api/v1/portal/add_age_verification_profile/
|
||||
/api/v1/portal/affiliate_count_increment/
|
||||
/api/v1/portal/alert_events/
|
||||
/api/v1/portal/chargebee_checkout_url/
|
||||
/api/v1/portal/chargebee_self_serve_portal_url/
|
||||
/api/v1/portal/child_activation_deleted/
|
||||
/api/v1/portal/child_graduated/
|
||||
/api/v1/portal/create_age_user/
|
||||
/api/v1/portal/create_temp_network_qr_code/
|
||||
/api/v1/portal/delete_age_user/
|
||||
/api/v1/portal/download_game_machine_audit/{audit_download_uid}/
|
||||
/api/v1/portal/events/
|
||||
/api/v1/portal/events/{pk}/
|
||||
/api/v1/portal/events/{pk}/children/
|
||||
/api/v1/portal/follow_user/
|
||||
/api/v1/portal/free_play_codes/
|
||||
/api/v1/portal/game_achivement_statistics/
|
||||
/api/v1/portal/game_layout_config/
|
||||
/api/v1/portal/game_machine_high_scores/
|
||||
/api/v1/portal/game_resources/
|
||||
/api/v1/portal/game_taxonomy_details/
|
||||
/api/v1/portal/game_teams/
|
||||
/api/v1/portal/get_age_permissions/
|
||||
/api/v1/portal/get_alerts_info_of_machines/
|
||||
/api/v1/portal/get_child_age/
|
||||
/api/v1/portal/home-leaderboard/{pk}/
|
||||
/api/v1/portal/launch-parties/
|
||||
/api/v1/portal/leaderboard-view-detail/{quest_id}/
|
||||
/api/v1/portal/leaderboard_events/
|
||||
/api/v1/portal/leaderboard_restricted/
|
||||
/api/v1/portal/leaderboard_url/{uuid}/
|
||||
/api/v1/portal/leaderboards/
|
||||
/api/v1/portal/leaderboards/machines/
|
||||
/api/v1/portal/leaderboards/titles/
|
||||
/api/v1/portal/leaderboards/{pk}/
|
||||
/api/v1/portal/machine_audits/
|
||||
/api/v1/portal/my_activity/
|
||||
/api/v1/portal/player_badges/
|
||||
/api/v1/portal/race_cups/
|
||||
/api/v1/portal/races/
|
||||
/api/v1/portal/resend_parent_email/
|
||||
/api/v1/portal/reset_password/
|
||||
/api/v1/portal/review_age_permissions/
|
||||
/api/v1/portal/set_all_access_pending/
|
||||
/api/v1/portal/shopify_all_access_membership_info/
|
||||
/api/v1/portal/shopify_all_access_purchase/
|
||||
/api/v1/portal/shopify_all_access_user_info/
|
||||
/api/v1/portal/shopify_claim_all_access_user_membership/
|
||||
/api/v1/portal/shopify_multipass_url/
|
||||
/api/v1/portal/sparks/
|
||||
/api/v1/portal/sso/
|
||||
/api/v1/portal/sso_logout/
|
||||
/api/v1/portal/support_code/
|
||||
/api/v1/portal/team_stern_members/
|
||||
/api/v1/portal/total_balls_played/
|
||||
/api/v1/portal/total_games_played/
|
||||
/api/v1/portal/track_page/
|
||||
/api/v1/portal/unfollow_user/
|
||||
/api/v1/portal/update_age_parent_email/
|
||||
/api/v1/portal/user_account_deleted/
|
||||
/api/v1/portal/user_acknowledge_notifications/
|
||||
/api/v1/portal/user_activities/
|
||||
/api/v1/portal/user_activities_from_session_details/
|
||||
/api/v1/portal/user_avatar/
|
||||
/api/v1/portal/user_badge_count/
|
||||
/api/v1/portal/user_badge_mark_read/
|
||||
/api/v1/portal/user_badges/
|
||||
/api/v1/portal/user_game_achievements/
|
||||
/api/v1/portal/user_game_locations/
|
||||
/api/v1/portal/user_game_networks/
|
||||
/api/v1/portal/user_game_overview/
|
||||
/api/v1/portal/user_highlights/
|
||||
/api/v1/portal/user_home_address_update/
|
||||
/api/v1/portal/user_login_qr_code/
|
||||
/api/v1/portal/user_notifications/
|
||||
/api/v1/portal/user_permission_changed/
|
||||
/api/v1/portal/user_registered_machines/
|
||||
/api/v1/portal/user_spark_rankings/
|
||||
/api/v1/portal/user_stats/
|
||||
/api/v1/portal/user_title_stats/
|
||||
/api/v1/portal/{webhook_secret}/
|
||||
/api/v1/privacy_types/
|
||||
/api/v1/province_codes/
|
||||
/api/v1/state_codes/
|
||||
/api/v1/time/
|
||||
/api/v1/time_zones/
|
||||
/api/v1/time_zones_for_country/
|
||||
/api/v1/url-config/
|
||||
/api/v2/auth_logout/
|
||||
/api/v2/game/alert_events/
|
||||
/api/v2/game/change_home_team/
|
||||
/api/v2/game/change_home_team_group/
|
||||
/api/v2/game/change_home_team_individual/
|
||||
/api/v2/game/free_play_code/
|
||||
/api/v2/game/game_achievement_descriptors/
|
||||
/api/v2/game/game_audit_descriptors/
|
||||
/api/v2/game/game_auth/
|
||||
/api/v2/game/game_machine_registration_removed/
|
||||
/api/v2/game/game_property_descriptors/
|
||||
/api/v2/game/heartbeat/
|
||||
/api/v2/game/high_score_events/
|
||||
/api/v2/game/machine_audits/
|
||||
/api/v2/game/payment_events/
|
||||
/api/v2/game/player_auth/
|
||||
/api/v2/game/player_list_info/
|
||||
/api/v2/game/player_properties/
|
||||
/api/v2/game/player_property/
|
||||
/api/v2/game/session_end/
|
||||
/api/v2/game/session_start/
|
||||
/api/v2/game/session_update/
|
||||
/api/v2/game/state/
|
||||
/api/v2/impersonate/
|
||||
/api/v2/portal/all_game_locations_with_games/
|
||||
/api/v2/portal/badges/
|
||||
/api/v2/portal/business_profile_confirm_registration/
|
||||
/api/v2/portal/business_profile_resend_registration_confirmation/
|
||||
/api/v2/portal/business_send_registration_invitation/
|
||||
/api/v2/portal/create_or_renew_all_access/
|
||||
/api/v2/portal/create_temp_network_qr_code/
|
||||
/api/v2/portal/game_get_hardware_removed_list/
|
||||
/api/v2/portal/game_location_players/
|
||||
/api/v2/portal/game_location_removal/
|
||||
/api/v2/portal/game_location_stats/
|
||||
/api/v2/portal/game_location_stats_by_group/
|
||||
/api/v2/portal/game_locations_search/
|
||||
/api/v2/portal/game_locations_with_games/
|
||||
/api/v2/portal/game_set_hardware_removed_status/
|
||||
/api/v2/portal/game_set_new_location/
|
||||
/api/v2/portal/game_set_online_status/
|
||||
/api/v2/portal/game_set_out_of_service_status/
|
||||
/api/v2/portal/home-leaderboards/
|
||||
/api/v2/portal/leaderboard_events/
|
||||
/api/v2/portal/quests/{pk}/
|
||||
/api/v2/portal/set_authd_password/
|
||||
/api/v2/portal/set_password/
|
||||
/api/v2/portal/user_completed_tutorial/
|
||||
/api/v2/portal/user_confirm_registration/
|
||||
/api/v2/portal/user_delete_account_request/
|
||||
/api/v2/portal/user_detail/
|
||||
/api/v2/portal/user_game_locations/
|
||||
/api/v2/portal/user_machine_registration_removed/
|
||||
/api/v2/portal/user_map_detail/
|
||||
/api/v2/portal/user_profile_detail/
|
||||
/api/v2/portal/user_profile_update/
|
||||
/api/v2/portal/user_registration_status/
|
||||
/api/v2/portal/user_set_privacy/
|
||||
/api/v2/portal/user_update/
|
||||
/api/v2/portal/user_verify_address/
|
||||
/api/v2/portal/user_verify_email/
|
||||
/api/v2/portal/user_verify_password/
|
||||
/api/v2/portal/user_verify_token_to_email/
|
||||
/api/v2/portal/user_verify_url/
|
||||
/api/v2/portal/user_verify_username/
|
||||
/api/v2/portal/validate_redemption_code/
|
||||
/api/v2/portal/verify_password_token/
|
||||
/api/v2/portal/wallet_webhook_token/
|
||||
/api/v2/remote_configs/
|
||||
/api/v2/token/
|
||||
/api/v2/token/refresh/
|
||||
/api/v3/auth_logout/
|
||||
/api/v3/game/game_achievement_descriptors/
|
||||
/api/v3/game/game_audit_descriptors/
|
||||
/api/v3/game/game_auth/
|
||||
/api/v3/game/game_machine_registration_removed/
|
||||
/api/v3/game/game_register/
|
||||
/api/v3/game/game_register_abort/
|
||||
/api/v3/game/player_auth/
|
||||
/api/v3/game/session_end/
|
||||
/api/v3/game/session_start/
|
||||
/api/v3/game/session_update/
|
||||
/api/v3/portal/
|
||||
/api/v3/portal/business_profile_register/
|
||||
/api/v3/portal/business_profile_update/
|
||||
/api/v3/portal/user_auth/
|
||||
/api/v3/portal/user_location_creation/
|
||||
/api/v3/portal/user_machine_registration/
|
||||
/api/v3/portal/user_register/
|
||||
/api/v3/portal/year-review/
|
||||
/api/v3/token/refresh/
|
||||
/api/v4/auth/login/
|
||||
/api/v4/auth/register/
|
||||
/api/v4/events/activity-near-user/
|
||||
/api/v4/events/get_quests_with_badges_for_event/
|
||||
/api/v4/events/launch-party/
|
||||
/api/v4/events/past_with_quests/
|
||||
/api/v4/events/with_quests/
|
||||
/api/v4/game/achievements/
|
||||
/api/v4/leaderboards/get_by_code/
|
||||
/api/v4/player/badges/
|
||||
/api/v4/player/info/
|
||||
/api/v4/player/is_public/
|
||||
/api/v4/player/model_scores/
|
||||
/api/v4/portal/klaviyo/
|
||||
/api/v4/portal/year-review/
|
||||
/api/v4/races/badges_with_descriptions/
|
||||
/api/v4/races/user_tier_for_race/
|
||||
/api/v4/recent/active_players/
|
||||
/api/v4/recent/badges/
|
||||
/api/v4/recent/high_scores/
|
||||
/api/v4/recent/played_games/
|
||||
/api/v4/recent/registered_machines/
|
||||
/api/v4/stats/achievement_completion_percentiles_by_title/
|
||||
/api/v4/stats/average_model_scores/
|
||||
/api/v4/stats/model_scores/
|
||||
/api/v4/stats/score_percentiles_by_title/
|
||||
/api/v4/stats/scores/
|
||||
/api/v4/stats/title/
|
||||
/api/v4/stats/user_xp_percentiles/
|
||||
/api/v4/user/achievements/
|
||||
/api/v4/user/activity/
|
||||
/api/v4/user/badges/
|
||||
/api/v4/user/create_location/
|
||||
/api/v4/user/followed_players/
|
||||
/api/v4/user/games-near-user/
|
||||
/api/v4/user/highlights/
|
||||
/api/v4/user/info/
|
||||
/api/v4/user/latest-achievements/
|
||||
/api/v4/user/locations_played_since_date/
|
||||
/api/v4/user/login_qr/
|
||||
/api/v4/user/milestones/
|
||||
/api/v4/user/notifications/
|
||||
/api/v4/user/quick_connect_qr/
|
||||
/api/v4/user/registered_machines/
|
||||
/api/v4/user/validate_address/
|
||||
/api/v4/user/wifi_qr/
|
||||
/webhook/pass/v1/devices/{device_id}/registrations/{pass_type_id}
|
||||
/webhook/pass/v1/devices/{device_id}/registrations/{pass_type_id}/{serial}
|
||||
/webhook/pass/v1/log
|
||||
/webhook/pass/v1/passes/{pass_type_id}/{serial}
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_DB: ${POSTGRES_DB}
|
||||
POSTGRES_USER: ${POSTGRES_USER}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
- ./db/init.sql:/docker-entrypoint-initdb.d/001-init.sql:ro
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
restart: unless-stopped
|
||||
|
||||
collector:
|
||||
build: .
|
||||
environment:
|
||||
API_BASE_URL: ${API_BASE_URL}
|
||||
API_USERNAME: ${API_USERNAME}
|
||||
API_PASSWORD: ${API_PASSWORD}
|
||||
POLL_SECONDS: ${POLL_SECONDS:-300}
|
||||
POSTGRES_HOST: postgres
|
||||
POSTGRES_PORT: 5432
|
||||
POSTGRES_DB: ${POSTGRES_DB}
|
||||
POSTGRES_USER: ${POSTGRES_USER}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana-oss:11.6.0
|
||||
ports:
|
||||
- "127.0.0.1:${GRAFANA_PORT:-3000}:3000"
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin}
|
||||
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
|
||||
GF_USERS_ALLOW_SIGN_UP: "false"
|
||||
POSTGRES_DB: ${POSTGRES_DB}
|
||||
POSTGRES_USER: ${POSTGRES_USER}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
volumes:
|
||||
- grafana-data:/var/lib/grafana
|
||||
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
grafana-data:
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,200 @@
|
||||
{
|
||||
"/api/v2/portal/user_confirm_registration/": {
|
||||
"bytes": 159,
|
||||
"contentType": "application/json",
|
||||
"schema": {
|
||||
"properties": {
|
||||
"authorized": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"code": {
|
||||
"type": "integer"
|
||||
},
|
||||
"datetime": {
|
||||
"type": "string"
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
},
|
||||
"message_number": {
|
||||
"type": "integer"
|
||||
},
|
||||
"success": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"status": 400
|
||||
},
|
||||
"/api/v2/portal/user_verify_email/": {
|
||||
"bytes": 150,
|
||||
"contentType": "application/json",
|
||||
"schema": {
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "integer"
|
||||
},
|
||||
"datetime": {
|
||||
"type": "string"
|
||||
},
|
||||
"email_present": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
},
|
||||
"message_number": {
|
||||
"type": "integer"
|
||||
},
|
||||
"success": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"status": 400
|
||||
},
|
||||
"/api/v2/portal/user_verify_password/": {
|
||||
"bytes": 244,
|
||||
"contentType": "application/json",
|
||||
"schema": {
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "integer"
|
||||
},
|
||||
"datetime": {
|
||||
"type": "string"
|
||||
},
|
||||
"errors": {
|
||||
"properties": {
|
||||
"validation_code": {
|
||||
"type": "integer"
|
||||
},
|
||||
"validation_message": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
},
|
||||
"message_number": {
|
||||
"type": "integer"
|
||||
},
|
||||
"password_present": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"success": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"status": 400
|
||||
},
|
||||
"/api/v2/portal/user_verify_username/": {
|
||||
"bytes": 156,
|
||||
"contentType": "application/json",
|
||||
"schema": {
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "integer"
|
||||
},
|
||||
"datetime": {
|
||||
"type": "string"
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
},
|
||||
"message_number": {
|
||||
"type": "integer"
|
||||
},
|
||||
"success": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"username_present": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"status": 400
|
||||
},
|
||||
"/api/v2/token/": {
|
||||
"bytes": 139,
|
||||
"contentType": "application/json",
|
||||
"schema": {
|
||||
"properties": {
|
||||
"password": {
|
||||
"items": {
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string"
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
"username": {
|
||||
"items": {
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string"
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"status": 400
|
||||
},
|
||||
"/api/v3/portal/user_auth/": {
|
||||
"bytes": 206,
|
||||
"contentType": "application/json",
|
||||
"schema": {
|
||||
"properties": {
|
||||
"authorized": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"code": {
|
||||
"type": "integer"
|
||||
},
|
||||
"country_of_request": {
|
||||
"nullable": true
|
||||
},
|
||||
"datetime": {
|
||||
"type": "string"
|
||||
},
|
||||
"identifier": {
|
||||
"type": "string"
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
},
|
||||
"message_number": {
|
||||
"type": "integer"
|
||||
},
|
||||
"success": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"username": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"status": 401
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
{
|
||||
"uid": "stern-play-history",
|
||||
"title": "Stern Play History",
|
||||
"tags": ["stern", "scores"],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "5m",
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "game_model_type",
|
||||
"label": "Game model type",
|
||||
"type": "query",
|
||||
"datasource": {"type": "postgres", "uid": "play-history-postgres"},
|
||||
"query": "SELECT DISTINCT game_model_type FROM activity_snapshot ORDER BY 1",
|
||||
"definition": "SELECT DISTINCT game_model_type FROM activity_snapshot ORDER BY 1",
|
||||
"includeAll": true,
|
||||
"multi": true,
|
||||
"current": {"text": "All", "value": ["$__all"]}
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Maximum score by observed activity",
|
||||
"type": "timeseries",
|
||||
"datasource": {"type": "postgres", "uid": "play-history-postgres"},
|
||||
"gridPos": {"h": 10, "w": 12, "x": 0, "y": 0},
|
||||
"targets": [
|
||||
{
|
||||
"format": "time_series",
|
||||
"rawSql": "SELECT observed_at AS \"time\", max_score AS \"max score\", game_model_type || ' #' || game_model_id AS metric FROM activity_snapshot WHERE $__timeFilter(observed_at) AND game_model_type IN (${game_model_type:sqlstring}) ORDER BY 1",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {"defaults": {"unit": "none"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "Total plays by observed activity",
|
||||
"type": "timeseries",
|
||||
"datasource": {"type": "postgres", "uid": "play-history-postgres"},
|
||||
"gridPos": {"h": 10, "w": 12, "x": 12, "y": 0},
|
||||
"targets": [
|
||||
{
|
||||
"format": "time_series",
|
||||
"rawSql": "SELECT observed_at AS \"time\", total_plays AS \"total plays\", game_model_type || ' #' || game_model_id AS metric FROM activity_snapshot WHERE $__timeFilter(observed_at) AND game_model_type IN (${game_model_type:sqlstring}) ORDER BY 1",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {"defaults": {"unit": "none"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Latest collected activity",
|
||||
"type": "table",
|
||||
"datasource": {"type": "postgres", "uid": "play-history-postgres"},
|
||||
"gridPos": {"h": 10, "w": 24, "x": 0, "y": 10},
|
||||
"targets": [
|
||||
{
|
||||
"format": "table",
|
||||
"rawSql": "SELECT DISTINCT ON (game_model_type, game_model_id, location_id) observed_at AS \"observed at\", activity_date AS \"activity date\", game_model_type, game_model_id, location_name, max_score, total_plays FROM activity_snapshot WHERE game_model_type IN (${game_model_type:sqlstring}) ORDER BY game_model_type, game_model_id, location_id, observed_at DESC",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: Play History
|
||||
orgId: 1
|
||||
folder: Play History
|
||||
type: file
|
||||
disableDeletion: true
|
||||
editable: false
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Play History
|
||||
uid: play-history-postgres
|
||||
type: postgres
|
||||
access: proxy
|
||||
url: postgres:5432
|
||||
user: $POSTGRES_USER
|
||||
secureJsonData:
|
||||
password: $POSTGRES_PASSWORD
|
||||
jsonData:
|
||||
database: $POSTGRES_DB
|
||||
sslmode: disable
|
||||
postgresVersion: 1600
|
||||
timescaledb: false
|
||||
isDefault: true
|
||||
editable: false
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,436 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build a best-effort OpenAPI manifest from Django DEBUG URL patterns and OPTIONS metadata.
|
||||
Discovery-only: sends GET/OPTIONS plus deliberately invalid credential-validation POSTs.
|
||||
"""
|
||||
|
||||
import concurrent.futures
|
||||
import html
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.error import HTTPError, URLError
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
BASE = "https://iop-dev.strndev.com"
|
||||
OUT = Path(__file__).parent
|
||||
UA = "stern-openapi-reconstruction/1.0 (read-only documentation)"
|
||||
|
||||
|
||||
def request(path, method="GET"):
|
||||
req = Request(
|
||||
BASE + path,
|
||||
method=method,
|
||||
headers={"Accept": "application/json", "User-Agent": UA},
|
||||
)
|
||||
try:
|
||||
with urlopen(req, timeout=20) as res:
|
||||
return res.status, dict(res.headers.items()), res.read()
|
||||
except HTTPError as e:
|
||||
return e.code, dict(e.headers.items()), e.read()
|
||||
except (URLError, OSError, TimeoutError) as e:
|
||||
return 0, {}, str(e).encode()
|
||||
|
||||
|
||||
def django_routes(page):
|
||||
# Django debug view puts individual patterns in li nodes.
|
||||
values = []
|
||||
for raw in re.findall(r"<li>(.*?)</li>", page, flags=re.DOTALL):
|
||||
value = re.sub(r"\s+", " ", html.unescape(re.sub(r"<[^>]+>", " ", raw))).strip()
|
||||
value = re.sub(r"\s*\[name=.*?\]$", "", value)
|
||||
if value.startswith(("api/", "webhook/")):
|
||||
values.append(value)
|
||||
return list(dict.fromkeys(values))
|
||||
|
||||
|
||||
def normalize(route):
|
||||
route = route.strip()
|
||||
route = route.removeprefix("^").removesuffix("$")
|
||||
route = route.replace("?", "")
|
||||
# Convert Django path converters to OAS template syntax.
|
||||
route = re.sub(
|
||||
r"<(?:(?:uuid|slug|str|int):)?([A-Za-z_][A-Za-z0-9_]*)>", r"{\1}", route
|
||||
)
|
||||
# Never copy externally exposed secret-looking webhook tokens into output.
|
||||
if re.search(r"/(?:[A-Za-z0-9]{24,})/?$", route) and "webhook" not in route:
|
||||
route = re.sub(r"/[A-Za-z0-9]{24,}/?$", "/{webhook_secret}/", route)
|
||||
return "/" + route.lstrip("/")
|
||||
|
||||
|
||||
def options(path):
|
||||
status, headers, body = request(path, "OPTIONS")
|
||||
lower = {k.lower(): v for k, v in headers.items()}
|
||||
try:
|
||||
data = json.loads(body)
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
data = None
|
||||
allow = [
|
||||
x.strip().lower()
|
||||
for x in lower.get("allow", "").split(",")
|
||||
if x.strip() and x.strip().upper() not in {"HEAD", "OPTIONS"}
|
||||
]
|
||||
auth = lower.get("www-authenticate")
|
||||
if isinstance(data, dict) and data.get("actions"):
|
||||
allow = list(dict.fromkeys(allow + [x.lower() for x in data["actions"]]))
|
||||
return {"status": status, "allow": allow, "auth": auth, "metadata": data}
|
||||
|
||||
|
||||
def merge_shapes(shapes: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
if not shapes:
|
||||
return {}
|
||||
types = {shape.get("type") for shape in shapes}
|
||||
if len(types) != 1:
|
||||
unique = {json.dumps(shape, sort_keys=True): shape for shape in shapes}
|
||||
return {"oneOf": list(unique.values())}
|
||||
result = dict(shapes[0])
|
||||
if result.get("type") == "object":
|
||||
keys = set().union(*(shape.get("properties", {}) for shape in shapes))
|
||||
result["properties"] = {
|
||||
key: merge_shapes(
|
||||
[
|
||||
shape["properties"][key]
|
||||
for shape in shapes
|
||||
if key in shape.get("properties", {})
|
||||
]
|
||||
)
|
||||
for key in sorted(keys)
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def json_shape(value: Any) -> dict[str, Any]:
|
||||
if isinstance(value, dict):
|
||||
return {
|
||||
"type": "object",
|
||||
"properties": {str(key): json_shape(item) for key, item in value.items()},
|
||||
}
|
||||
if isinstance(value, list):
|
||||
return {
|
||||
"type": "array",
|
||||
"items": merge_shapes([json_shape(item) for item in value]),
|
||||
}
|
||||
if isinstance(value, bool):
|
||||
return {"type": "boolean"}
|
||||
if isinstance(value, int):
|
||||
return {"type": "integer"}
|
||||
if isinstance(value, float):
|
||||
return {"type": "number"}
|
||||
if value is None:
|
||||
return {"nullable": True}
|
||||
return {"type": "string"}
|
||||
|
||||
|
||||
def schema_example(schema: dict[str, Any]) -> Any:
|
||||
if "oneOf" in schema:
|
||||
return schema_example(schema["oneOf"][0])
|
||||
if schema.get("type") == "object":
|
||||
return {
|
||||
name: schema_example(value)
|
||||
for name, value in schema.get("properties", {}).items()
|
||||
}
|
||||
if schema.get("type") == "array":
|
||||
return [schema_example(schema.get("items", {}))]
|
||||
primitive_examples = {
|
||||
"boolean": False,
|
||||
"integer": 0,
|
||||
"number": 0,
|
||||
"string": "string",
|
||||
}
|
||||
schema_type = schema.get("type")
|
||||
return primitive_examples.get(schema_type) if isinstance(schema_type, str) else None
|
||||
|
||||
|
||||
def post_validation_evidence(path: str) -> dict[str, Any]:
|
||||
req = Request(
|
||||
BASE + path,
|
||||
data=b"{}",
|
||||
method="POST",
|
||||
headers={
|
||||
"Accept": "application/json",
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": UA,
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urlopen(req, timeout=20) as res:
|
||||
status, headers, body = res.status, dict(res.headers.items()), res.read()
|
||||
except HTTPError as exc:
|
||||
status, headers, body = exc.code, dict(exc.headers.items()), exc.read()
|
||||
except (URLError, OSError, TimeoutError) as exc:
|
||||
return {"status": 0, "error": str(exc)}
|
||||
result: dict[str, Any] = {
|
||||
"status": status,
|
||||
"contentType": headers.get("Content-Type", ""),
|
||||
"bytes": len(body),
|
||||
}
|
||||
if "json" in result["contentType"].lower():
|
||||
try:
|
||||
result["schema"] = json_shape(json.loads(body))
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
result["jsonParseError"] = True
|
||||
return result
|
||||
|
||||
|
||||
def get_evidence(path: str) -> dict[str, Any]:
|
||||
status, headers, body = request(path)
|
||||
content_type = headers.get("Content-Type", "")
|
||||
result: dict[str, Any] = {
|
||||
"status": status,
|
||||
"contentType": content_type,
|
||||
"bytes": len(body),
|
||||
}
|
||||
if 200 <= status < 300 and "json" in content_type.lower():
|
||||
try:
|
||||
result["schema"] = json_shape(json.loads(body))
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
result["jsonParseError"] = True
|
||||
return result
|
||||
|
||||
|
||||
def field_schema(fields: dict[str, Any]) -> dict[str, Any]:
|
||||
props: dict[str, Any] = {}
|
||||
required: list[str] = []
|
||||
mapping = {
|
||||
"string": "string",
|
||||
"integer": "integer",
|
||||
"boolean": "boolean",
|
||||
"number": "number",
|
||||
"field": "object",
|
||||
"choice": "string",
|
||||
"file upload": "string",
|
||||
}
|
||||
for name, field in fields.items():
|
||||
if not isinstance(field, dict):
|
||||
continue
|
||||
schema: dict[str, Any] = {
|
||||
"type": mapping.get(field.get("type", "string"), "string")
|
||||
}
|
||||
if field.get("label"):
|
||||
schema["title"] = field["label"]
|
||||
if field.get("read_only"):
|
||||
schema["readOnly"] = True
|
||||
if field.get("write_only"):
|
||||
schema["writeOnly"] = True
|
||||
if field.get("choices"):
|
||||
schema["enum"] = [
|
||||
c.get("value", c) if isinstance(c, dict) else c
|
||||
for c in field["choices"]
|
||||
]
|
||||
props[name] = schema
|
||||
if field.get("required"):
|
||||
required.append(name)
|
||||
result = {"type": "object", "properties": props}
|
||||
if required:
|
||||
result["required"] = required
|
||||
return result
|
||||
|
||||
|
||||
def path_parameters(path):
|
||||
return [
|
||||
{
|
||||
"name": name,
|
||||
"in": "path",
|
||||
"required": True,
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"format": "uuid" if name in {"audit_download_uid", "uuid"} else None,
|
||||
},
|
||||
}
|
||||
for name in re.findall(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", path)
|
||||
]
|
||||
|
||||
|
||||
def operation(
|
||||
method: str,
|
||||
evidence: dict[str, Any],
|
||||
get_result: dict[str, Any] | None,
|
||||
validation_result: dict[str, Any] | None,
|
||||
) -> dict[str, Any]:
|
||||
metadata, auth = evidence.get("metadata"), evidence.get("auth")
|
||||
success: dict[str, Any] = {"description": "Success (shape not fully inferred)"}
|
||||
if method == "get" and get_result and "schema" in get_result:
|
||||
success = {
|
||||
"description": "Observed public response shape; field optionality and array item variance are not inferred.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": get_result["schema"],
|
||||
"example": schema_example(get_result["schema"]),
|
||||
}
|
||||
},
|
||||
}
|
||||
op: dict[str, Any] = {
|
||||
"responses": {
|
||||
"200": success,
|
||||
"401": {"$ref": "#/components/responses/Unauthorized"},
|
||||
"403": {"$ref": "#/components/responses/Forbidden"},
|
||||
}
|
||||
}
|
||||
if method == "post" and validation_result and "schema" in validation_result:
|
||||
op["responses"][str(validation_result["status"])] = {
|
||||
"description": "Observed validation response to an empty JSON object.",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": validation_result["schema"],
|
||||
"example": schema_example(validation_result["schema"]),
|
||||
}
|
||||
},
|
||||
}
|
||||
if isinstance(metadata, dict):
|
||||
if metadata.get("name"):
|
||||
op["summary"] = metadata["name"]
|
||||
if metadata.get("description"):
|
||||
op["description"] = metadata["description"]
|
||||
fields = metadata.get("actions", {}).get(method.upper())
|
||||
if isinstance(fields, dict):
|
||||
op["requestBody"] = {
|
||||
"required": True,
|
||||
"content": {
|
||||
ctype: {
|
||||
"schema": field_schema(fields),
|
||||
"example": schema_example(field_schema(fields)),
|
||||
}
|
||||
for ctype in metadata.get("parses", ["application/json"])
|
||||
},
|
||||
}
|
||||
if auth:
|
||||
op["security"] = [{"bearerAuth": []}]
|
||||
return op
|
||||
|
||||
|
||||
def main():
|
||||
status, _, debug = request("/openapi-reconstruction-does-not-exist")
|
||||
if status != 404:
|
||||
raise SystemExit(f"Expected Django debug 404; received {status}")
|
||||
page = debug.decode("utf-8", "replace")
|
||||
raw_routes = django_routes(page)
|
||||
paths = sorted({normalize(route) for route in raw_routes})
|
||||
# Retain only normalized paths: the raw DEBUG page contains opaque webhook URLs.
|
||||
(OUT / "django-404-urlpatterns.routes.txt").write_text("\n".join(paths) + "\n")
|
||||
|
||||
# OPTIONS is a safe discovery method but is throttled to five concurrent requests.
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as pool:
|
||||
records = dict(zip(paths, pool.map(options, paths), strict=True))
|
||||
(OUT / "endpoint-options-evidence.json").write_text(
|
||||
json.dumps(records, indent=2, sort_keys=True, default=str)
|
||||
)
|
||||
readable_paths = [
|
||||
path for path, record in records.items() if "get" in record["allow"]
|
||||
]
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as pool:
|
||||
get_records = dict(
|
||||
zip(readable_paths, pool.map(get_evidence, readable_paths), strict=True)
|
||||
)
|
||||
(OUT / "endpoint-get-evidence.json").write_text(
|
||||
json.dumps(get_records, indent=2, sort_keys=True)
|
||||
)
|
||||
# These endpoints only validate credentials/registration data; `{}` is deliberately invalid and cannot complete their action.
|
||||
validation_paths = [
|
||||
"/api/v2/token/",
|
||||
"/api/v2/portal/user_verify_email/",
|
||||
"/api/v2/portal/user_verify_password/",
|
||||
"/api/v2/portal/user_verify_username/",
|
||||
"/api/v2/portal/user_confirm_registration/",
|
||||
"/api/v3/portal/user_auth/",
|
||||
]
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
|
||||
validation_records = dict(
|
||||
zip(
|
||||
validation_paths,
|
||||
pool.map(post_validation_evidence, validation_paths),
|
||||
strict=True,
|
||||
)
|
||||
)
|
||||
(OUT / "endpoint-validation-evidence.json").write_text(
|
||||
json.dumps(validation_records, indent=2, sort_keys=True)
|
||||
)
|
||||
|
||||
manifest = {
|
||||
"openapi": "3.0.3",
|
||||
"info": {
|
||||
"title": "Stern Insider (iop-dev) reconstructed API",
|
||||
"version": "0.1.0-reconstructed",
|
||||
"description": "Best-effort manifest reconstructed from the public Django DEBUG URLconf and read-only OPTIONS responses. It is not an authoritative contract; operations without OPTIONS metadata have inferred methods only.",
|
||||
},
|
||||
"servers": [{"url": BASE}],
|
||||
"paths": {},
|
||||
"components": {
|
||||
"securitySchemes": {
|
||||
"bearerAuth": {
|
||||
"type": "http",
|
||||
"scheme": "bearer",
|
||||
"bearerFormat": "JWT",
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"Unauthorized": {
|
||||
"description": "Authentication credentials were not provided or are invalid."
|
||||
},
|
||||
"Forbidden": {"description": "Authenticated caller lacks permission."},
|
||||
},
|
||||
},
|
||||
"x-reconstruction": {
|
||||
"generatedAt": datetime.now(timezone.utc).isoformat(),
|
||||
"source": "Public Django DEBUG 404 URLconf plus OPTIONS metadata",
|
||||
"routeCount": len(paths),
|
||||
"rawRouteCount": len(raw_routes),
|
||||
"limitations": [
|
||||
"Response schemas were not inferred unless supplied by the server.",
|
||||
"OPTIONS may be denied before serializer metadata is exposed.",
|
||||
"Secret-looking webhook path tokens are redacted as {webhook_secret}.",
|
||||
"Non-api HTML routes are intentionally excluded; see API_RECONSTRUCTION.md.",
|
||||
],
|
||||
},
|
||||
}
|
||||
for path in paths:
|
||||
evidence = records[path]
|
||||
methods = evidence["allow"]
|
||||
item: dict[str, Any] = {
|
||||
m: operation(
|
||||
m, evidence, get_records.get(path), validation_records.get(path)
|
||||
)
|
||||
for m in methods
|
||||
}
|
||||
parameters = path_parameters(path)
|
||||
for parameter in parameters:
|
||||
parameter["schema"] = {
|
||||
key: value
|
||||
for key, value in parameter["schema"].items()
|
||||
if value is not None
|
||||
}
|
||||
if parameters:
|
||||
item["parameters"] = parameters
|
||||
item["x-options-evidence"] = {
|
||||
"status": evidence["status"],
|
||||
"auth": evidence["auth"],
|
||||
"hasSerializerMetadata": bool(
|
||||
isinstance(evidence["metadata"], dict)
|
||||
and evidence["metadata"].get("actions")
|
||||
),
|
||||
}
|
||||
if not methods:
|
||||
item["x-reconstruction-status"] = (
|
||||
"Route is exposed in the Django URLconf, but no operation was inferred because OPTIONS returned no Allow header."
|
||||
)
|
||||
manifest["paths"][path] = item
|
||||
(OUT / "openapi.reconstructed.json").write_text(
|
||||
json.dumps(manifest, indent=2) + "\n"
|
||||
)
|
||||
http_methods = {"get", "post", "put", "patch", "delete", "head", "options", "trace"}
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"routes": len(paths),
|
||||
"operationsDocumented": sum(
|
||||
len([k for k in p if k in http_methods])
|
||||
for p in manifest["paths"].values()
|
||||
),
|
||||
"pathsWithOperations": len(manifest["paths"]),
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1 @@
|
||||
psycopg[binary]==3.2.9
|
||||
Reference in New Issue
Block a user