commit ef9bb21aaabc5bde89a1d94dbf476e704ab73c83
parent 4d8c975f9b6c9c457db5af251b4da03a30c1a197
Author: Erik Loualiche <eloualiche@users.noreply.github.com>
Date: Sun, 1 Mar 2026 22:05:40 -0600
Merge pull request #12 from LouLouLibs/demo/vhs-tape
Add VHS demo with credential redaction and redesign README
Diffstat:
6 files changed, 290 insertions(+), 29 deletions(-)
diff --git a/.gitignore b/.gitignore
@@ -4,6 +4,9 @@ wrds-dl-*
*.csv
.claude/
+# Demo (raw GIF contains unredacted credentials)
+demo-wrds-download-raw.gif
+
# Python
python/.venv/
python/uv.lock
diff --git a/Makefile b/Makefile
@@ -0,0 +1,13 @@
+.PHONY: demo demo-record demo-redact
+
+# Record the VHS demo and blur the username in one step.
+demo: demo-record demo-redact
+
+# Step 1: Record the raw GIF with VHS (outputs demo-wrds-download-raw.gif).
+demo-record:
+ vhs demo-wrds-download.tape
+
+# Step 2: Blur the username on the login screen (raw -> clean).
+# Override blur coordinates: BLUR_X=500 BLUR_Y=280 make demo-redact
+demo-redact:
+ ./scripts/redact-demo.sh demo-wrds-download-raw.gif demo-wrds-download.gif
diff --git a/README.md b/README.md
@@ -1,4 +1,7 @@
+<div align="center">
+
# wrds-dl
+### Browse and download WRDS data from your terminal
[](https://github.com/LouLouLibs/wrds-download/actions/workflows/ci.yml)
[](https://github.com/LouLouLibs/wrds-download/actions/workflows/ci-python.yml)
@@ -6,9 +9,25 @@
[](python/)
[](LICENSE)
-A terminal tool for browsing and downloading data from the [WRDS](https://wrds-www.wharton.upenn.edu/) PostgreSQL database. Output is Parquet or CSV.
+<img src="./demo-wrds-download.gif" width="800" alt="Demo of the TUI browsing WRDS schemas and downloading data">
+
+</div>
+
+---
+
+## Features
+
+- **Interactive TUI** — browse schemas, tables, and column metadata with keyboard navigation
+- **CLI download** — script-friendly `download` and `info` commands for automation and HPC
+- **Parquet & CSV** — output to compressed Parquet (ZSTD) or CSV
+- **Dry run** — preview queries, row counts, and sample rows before downloading
+- **Raw SQL** — run arbitrary queries when schema/table syntax isn't enough
+- **Streaming** — server-side cursors keep memory usage low on large tables
+- **Saved credentials** — authenticate once, connect instantly
+
+## Two implementations
-Two implementations with the same CLI interface — pick whichever fits your environment:
+Same CLI interface — pick whichever fits your environment:
| | [Go](go/) | [Python](python/) |
|---|---|---|
@@ -41,7 +60,7 @@ uv tool install wrds-dl --from ./python
cd python && uv run wrds-dl --help
```
-## CLI (both implementations)
+## Usage
```sh
# CRSP monthly stock file — prices and returns for 2020
@@ -57,12 +76,7 @@ wrds-dl info --schema crsp --table msf
wrds-dl download --schema crsp --table msf \
--where "date = '2020-01-31'" --dry-run
-# CRSP daily stock file
-wrds-dl download --schema crsp --table dsf \
- --where "date >= '2020-01-01' AND date < '2021-01-01'" \
- --out crsp_dsf_2020.parquet
-
-# Select specific columns from Compustat
+# Compustat fundamentals
wrds-dl download --schema comp --table funda \
--columns "gvkey,datadate,sale,at" \
--out funda_subset.parquet
@@ -78,24 +92,6 @@ wrds-dl download --schema crsp --table msf \
--out crsp_msf_sample.csv
```
-## Claude Code skill
-
-Bundled [Claude Code](https://claude.com/claude-code) skills let you download WRDS data using natural language:
-
-```
-/wrds-download CRSP daily stock data for 2020
-```
-
-Two variants available:
-- [`claude-skill-wrds-download/`](claude-skill-wrds-download/) — uses the Go binary
-- [`claude-skill-wrds-download-py/`](claude-skill-wrds-download-py/) — uses the Python CLI (no binary needed)
-
-Install by copying the skill into your skills directory:
-
-```sh
-cp -r claude-skill-wrds-download-py ~/.claude/skills/wrds-download
-```
-
## Authentication
WRDS uses Duo two-factor authentication. Configure credentials before using the CLI.
@@ -117,7 +113,7 @@ export PGDATABASE=wrds
### Option 2: Saved credentials
-Store credentials at `~/.config/wrds-dl/credentials` (or `$XDG_CONFIG_HOME/wrds-dl/credentials`) with `0600` permissions:
+Store at `~/.config/wrds-dl/credentials` (or `$XDG_CONFIG_HOME/wrds-dl/credentials`) with `0600` permissions:
```
PGUSER=your_username
@@ -125,7 +121,7 @@ PGPASSWORD=your_password
PGDATABASE=wrds
```
-The Go TUI can save these automatically on first login.
+The Go TUI saves these automatically on first login.
### Option 3: ~/.pgpass
@@ -135,6 +131,24 @@ Standard PostgreSQL password file:
wrds-pgdata.wharton.upenn.edu:9737:*:your_username:your_password
```
+## Claude Code skill
+
+Bundled [Claude Code](https://claude.com/claude-code) skills let you download WRDS data using natural language:
+
+```
+/wrds-download CRSP daily stock data for 2020
+```
+
+Two variants available:
+- [`claude-skill-wrds-download/`](claude-skill-wrds-download/) — uses the Go binary
+- [`claude-skill-wrds-download-py/`](claude-skill-wrds-download-py/) — uses the Python CLI (no binary needed)
+
+Install by copying the skill into your skills directory:
+
+```sh
+cp -r claude-skill-wrds-download-py ~/.claude/skills/wrds-download
+```
+
## Project structure
```
@@ -150,3 +164,7 @@ wrds-download/
├── claude-skill-wrds-download-py/ # Claude skill (Python/uv)
└── .github/workflows/ # CI for both implementations
```
+
+## License
+
+MIT
diff --git a/demo-wrds-download.gif b/demo-wrds-download.gif
Binary files differ.
diff --git a/demo-wrds-download.tape b/demo-wrds-download.tape
@@ -0,0 +1,167 @@
+# VHS tape for recording wrds-dl TUI demo GIF.
+# Usage: vhs demo-wrds-download.tape
+#
+# Prerequisites:
+# - wrds-dl binary built and in PATH (or ./go/wrds-dl)
+# - WRDS credentials saved (~/.config/wrds-dl/credentials)
+# or exported as PGUSER / PGPASSWORD environment variables
+# - Active network connection to WRDS (Duo 2FA approved)
+
+Output demo-wrds-download-raw.gif
+
+Set FontSize 18
+Set Width 1400
+Set Height 700
+Set Padding 20
+Set Theme "GruvboxDarkHard"
+
+Set TypingSpeed 80ms
+
+Set Shell "bash"
+
+Hide
+ Type 'export PS1="> "'
+ Enter
+ Type "clear"
+ Enter
+Show
+
+# ─── 1. LAUNCH THE TUI ───
+Type "./wrds-dl tui"
+Enter
+
+# Wait for login screen to render
+Sleep 2s
+
+# ─── 2. LOGIN WITH SAVED CREDENTIALS ───
+# The login form shows "Login as <user> [enter]" when credentials exist.
+# Press Enter to connect with saved credentials.
+Enter
+
+# Wait for connection + Duo 2FA + schemas to load
+Sleep 2s
+
+# ─── 3. BROWSE SCHEMAS ───
+# Schemas pane is focused by default. Navigate to find "crsp".
+Sleep 500ms
+Down 1
+Sleep 300ms
+Down 1
+Sleep 300ms
+Down 1
+Sleep 300ms
+Down 1
+Sleep 300ms
+Down 1
+Sleep 1s
+
+# Use "/" to filter schemas
+Type "/"
+Sleep 300ms
+Type "crsp"
+Sleep 1s
+Enter
+Sleep 1s
+
+# ─── 4. DRILL INTO SCHEMA ───
+# Press right arrow to select the schema and load its tables.
+Right
+Sleep 2s
+
+# ─── 5. BROWSE TABLES ───
+# Tables pane is now focused. Filter for "msf" (monthly stock file).
+Type "/"
+Sleep 300ms
+Type "msf"
+Sleep 1s
+Enter
+Sleep 2s
+
+# ─── 6. VIEW TABLE METADATA ───
+# Press right to drill into the table and see the preview pane.
+Right
+Sleep 3s
+
+# ─── 7. SCROLL THROUGH COLUMNS ───
+# Preview pane shows column catalog. Scroll with j/k.
+Sleep 500ms
+Down 1
+Sleep 300ms
+Down 1
+Sleep 300ms
+Down 1
+Sleep 300ms
+Down 1
+Sleep 300ms
+Down 1
+Sleep 1s
+
+# Scroll back up
+Up 1
+Sleep 200ms
+Up 1
+Sleep 200ms
+Up 1
+Sleep 1s
+
+# ─── 8. FILTER COLUMNS ───
+# Press "/" in the preview pane to filter columns.
+Type "/"
+Sleep 300ms
+Type "ret"
+Sleep 1s
+# Press Escape to clear the filter
+Escape
+Sleep 1s
+
+# ─── 9. OPEN DOWNLOAD DIALOG ───
+# Press "d" to open the download form.
+Type "d"
+Sleep 1s
+
+# ─── 10. FILL IN DOWNLOAD FORM ───
+# SELECT columns field — clear prepopulated "*" then type columns
+Ctrl+U
+Sleep 200ms
+Type "permno,date,prc,ret,shrout"
+Sleep 500ms
+
+# Move to WHERE clause
+Tab
+Sleep 300ms
+Type "date >= '2024-01-01' AND date < '2024-02-01'"
+Sleep 500ms
+
+# Move to LIMIT — leave empty (no limit)
+Tab
+Sleep 300ms
+
+# Move to Output path
+Tab
+Sleep 300ms
+# Clear prepopulated path then type custom path
+Ctrl+U
+Sleep 200ms
+Type "./crsp_msf_2024_jan.parquet"
+Sleep 500ms
+
+# Move to Format — keep default "parquet"
+Tab
+Sleep 1s
+
+# ─── 11. SUBMIT DOWNLOAD ───
+Enter
+Sleep 5s
+
+# ─── 12. DISMISS SUCCESS MESSAGE ───
+Escape
+Sleep 1s
+
+# ─── 13. QUIT ───
+Type "q"
+Sleep 500ms
+
+# ─── 14. VERIFY OUTPUT ───
+Type "du -h crsp_msf_2024_jan.parquet"
+Enter
+Sleep 3s
diff --git a/scripts/redact-demo.sh b/scripts/redact-demo.sh
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# redact-demo.sh — Blur the WRDS username on the login screen of a VHS-recorded GIF.
+#
+# Usage:
+# ./scripts/redact-demo.sh # default paths
+# ./scripts/redact-demo.sh raw.gif clean.gif # explicit paths
+#
+# All blur parameters can be overridden via environment variables.
+# Record once, inspect the raw GIF, then tweak BLUR_X/Y/W/H if needed.
+#
+# Requires: ffmpeg
+
+set -euo pipefail
+
+INPUT="${1:-demo-wrds-download-raw.gif}"
+OUTPUT="${2:-demo-wrds-download.gif}"
+
+if [[ ! -f "$INPUT" ]]; then
+ echo "Error: $INPUT not found. Run 'vhs demo-wrds-download.tape' first." >&2
+ exit 1
+fi
+
+if ! command -v ffmpeg &>/dev/null; then
+ echo "Error: ffmpeg is required. Install with: brew install ffmpeg" >&2
+ exit 1
+fi
+
+# ── Blur region (pixels) ──
+# These cover the username text inside the "Login as <user> [enter]" button.
+# Calibrated from the login frame at t≈2s (1400x700, FontSize 18, Padding 20).
+BLUR_X="${BLUR_X:-490}"
+BLUR_Y="${BLUR_Y:-210}"
+BLUR_W="${BLUR_W:-100}"
+BLUR_H="${BLUR_H:-20}"
+
+# ── Time window (seconds) ──
+# The login screen is visible from launch until schemas load.
+BLUR_START="${BLUR_START:-0}"
+BLUR_END="${BLUR_END:-5}"
+
+# ── Pixelation factor (higher = more pixelated, less readable) ──
+PIXEL_FACTOR="${PIXEL_FACTOR:-10}"
+
+TEMP=$(mktemp -d)
+trap 'rm -rf "$TEMP"' EXIT
+
+echo "Pixelating region (${BLUR_X},${BLUR_Y}) ${BLUR_W}x${BLUR_H} during t=${BLUR_START}s-${BLUR_END}s (factor=${PIXEL_FACTOR})..."
+
+ffmpeg -y -loglevel error -i "$INPUT" -filter_complex \
+ "[0:v]crop=${BLUR_W}:${BLUR_H}:${BLUR_X}:${BLUR_Y}, \
+ scale=iw/${PIXEL_FACTOR}:ih/${PIXEL_FACTOR}, \
+ scale=${BLUR_W}:${BLUR_H}:flags=neighbor[px]; \
+ [0:v][px]overlay=${BLUR_X}:${BLUR_Y}:enable='between(t,${BLUR_START},${BLUR_END})'[v]; \
+ [v]split[s0][s1]; \
+ [s0]palettegen=max_colors=256:stats_mode=diff[p]; \
+ [s1][p]paletteuse=dither=bayer:bayer_scale=3" \
+ "$TEMP/out.gif"
+
+mv "$TEMP/out.gif" "$OUTPUT"
+echo "Done: $INPUT -> $OUTPUT"
