Code Embedding Lab

An English-only educational experiment comparing embeddings of source code and natural-language descriptions.

Contents

• samples/: Python, C#, x86-64 NASM and English-language descriptions.
• cases.json: all 72 pairs, including positive, negative, and partial expectations.
• asm_cases.json: 28 assembly-related comparisons.
• embedding_similarity.py: similarity(phrase1, phrase2, token) -> float.
• runner.py: executes comparisons and saves results in results.csv.

Important: Code sample filenames, Python function names, C# method/class names, and NASM exported symbols have been replaced with meaningless identifiers. The runner sends only file contents, not filenames, notes, labels, or expected outcomes, to the embedding API. Metadata is retained in the test-case JSON for evaluation.

Run (PowerShell)

python -m pip install -r requirements.txt
$env:OPENROUTER_API_KEY = "your_key_here"
python runner.py --limit 5
python runner.py
python runner.py --cases asm_cases.json --limit 5

On Linux/macOS set export OPENROUTER_API_KEY="your_key_here" instead.

Each comparison makes an API request. Check provider usage limits before a full run. The key is not saved in the CSV file.

Interpreting results

• positive: related purpose or algorithms, not proof of identical behavior on every input.
• negative: unrelated tasks or descriptions.
• partial: similar purpose but distinct behavior, such as exact-byte versus case-insensitive palindrome checks.

Some cross-language cases differ in edge behavior, including Unicode normalization, signed inputs, and integer overflow. A universal cosine threshold such as 0.95 cannot be assumed without calibration.

The assembly examples use NASM x86-64 System V calling conventions and are functions intended to be linked into another program, not standalone executables. Windows x64 has a different calling convention. Assembly correctness and actual API responses have not been independently validated.

Experimental limitation

Although function names and filenames are opaque, variable names, built-in operations, and code structure can still reveal semantics. This experiment measures similarity without meaningful function names, not complete resistance to lexical clues. The supplied descriptions remain meaningful in English by design.