Hackathon: Benchmarking Small Language Models in the Real World

Judging Criteria

The evaluation is designed specifically for this hackathon and reflects what matters most for the task: building the best text-to-Polars query generation model.

Submissions are evaluated primarily on their ability to generate correct Polars queries from natural language prompts. The most important factor is therefore the number of correct answers produced on the evaluation set.

Leaderboard scoring (Global submissions)

The public leaderboard ranks teams by their best completed Global submission. Scores come from the official benchmark endpoint (POST /submit_final), and higher is better.

Score = N / (T * VRAM^0.1 * RAM^0.01)

Where:

• N = number of correct answers (exact matches)
• T = total generation time across the evaluation set
• VRAM = GPU memory usage
• RAM = system memory usage

This reflects the order of importance:

1. Correctness (maximize N) — most important
2. Generation speed (minimize T)
3. VRAM efficiency (minimize VRAM)
4. RAM efficiency (minimize RAM; least important)

All teams are evaluated using the same set of benchmark questions and consistent evaluation conditions so results are directly comparable.

Tie-breakers (if scores are very close)

If two projects receive very similar scores, judges may use submission quality as a tie-breaker (reproducibility, clarity, documentation), with correctness remaining the primary objective.

Test runs vs Final submissions (important distinction)

The portal supports two submission modes:

Test submissions (test)

Use these to iterate quickly and debug. Test runs provide development feedback to help you improve correctness and performance, but may not match final leaderboard rankings.

Global submissions (global)

These are the submissions that count for the leaderboard and produce a Final Score using the formula above.

Best practices to score well

1) Maximize correctness (N)

• Start by getting correct outputs reliably on test runs before optimizing performance.
• Add regression tests and handle edge cases (nulls, empty frames, dtypes, parsing).
• Prefer deterministic logic and explicit error handling over “best-effort” fallbacks.

2) Reduce total time (T)

• Avoid Python loops over rows; prefer Polars expressions / vectorized operations.
• Reduce unnecessary joins/sorts; drop unused columns early.
• Cache repeated sub-computations where safe; avoid recomputing expensive intermediates.

3) Reduce VRAM and RAM

• Keep models small and efficient where possible (quantization / smaller checkpoints if it doesn’t hurt correctness).
• Avoid loading multiple model copies.
• Free large intermediates and minimize peak allocations during generation and execution.

4) Make your submission reproducible

• Provide a clear README with exact install and run steps.
• Pin dependencies (lockfiles / fixed versions).
• Ensure it runs non-interactively (no prompts), with sensible defaults and clear logs.

Submission checklist

• Repository URL is correct and accessible to the runner
• Your project runs end-to-end without manual steps
• A Test submission completes successfully
• A Global submission completes and shows a non-null Final Score
• README includes setup/run instructions and key assumptions