Issues

• Documentation: <|endoftext|> cannot be parsed unless encode function given extra parameters, this is not documented anywhere

  dqbd/tiktoken#155 · Filter by author aeisenberg opened on May 25, 2026
  2

• Add "bun" export condition so standalone bun compile can embed tiktoken_bg.wasm

  dqbd/tiktoken#154 · Filter by author CastIronPlatypus opened on Apr 19, 2026
  1

• OpenAI Harmony tokenzier

  dqbd/tiktoken#152 · Filter by author vladfaust opened on Nov 5, 2025

• [Feature] Add benchmarks (wasm vs js implementation). Benchmark here

  dqbd/tiktoken#147 · Filter by author kravetsone opened on May 23, 2025

• Not compatable with iife due to top-level awaits

  dqbd/tiktoken#145 · Filter by author douglasg14b opened on May 9, 2025

• Fetch request inside library could not be completed from some locations

  dqbd/tiktoken#138 · Filter by author opravdin opened on Mar 12, 2025
  1

• not able to find: text-embedding-3-small | npm module | "^1.0.18"

  dqbd/tiktoken#127 · Filter by author MwSpaceLLC opened on Feb 1, 2025
  3

• recursive use of an object detected which would lead to unsafe aliasing in rust

  dqbd/tiktoken#119 · Filter by author mogery opened on Aug 22, 2024
  2

• React native error SyntaxError: Invalid RegExp: Invalid escape] in react native expo.

  dqbd/tiktoken#110 · Filter by author djaffer opened on Jul 16, 2024
  1

• missing example code for js-tiktoken

  dqbd/tiktoken#102 · Filter by author coding22 opened on May 9, 2024
  1

• Performance

  dqbd/tiktoken#99 · Filter by author steebchen opened on Apr 15, 2024

• Add decode_with_offsets function (or PR guidance request)

  dqbd/tiktoken#94 · Filter by author Sam1320 opened on Mar 11, 2024

• Cloudflare worker Uncaught ReferenceError: FinalizationRegistry is not defined

  dqbd/tiktoken#92 · Filter by author TranquilMarmot opened on Feb 13, 2024
  1

• 'TextEncoder' is not defined - MongoDB Functions

  dqbd/tiktoken#91 · Filter by author georgeherby opened on Feb 11, 2024
  1

• Add support for the newest Open AI embeddings models

  dqbd/tiktoken#90 · Filter by author AleksTrpkovski opened on Feb 9, 2024
  2

• Memory access out of bounds

  dqbd/tiktoken#84 · Filter by author maxdeichmann opened on Jan 17, 2024
  3

• Crash on React Native: SyntaxError: Invalid RegExp: Invalid escape

  dqbd/tiktoken#82 · Filter by author julian-hecker opened on Dec 1, 2023
  2

• How to count tokens for functions defined in the prompt?

  dqbd/tiktoken#78 · Filter by author vrde opened on Oct 21, 2023
  1

• Support fine tunned models

  dqbd/tiktoken#76 · Filter by author italojs opened on Oct 17, 2023
  1

• Cannot find module error

  dqbd/tiktoken#75 · Filter by author thdoan opened on Oct 17, 2023
  1

• What does free actually do?

  dqbd/tiktoken#72 · Filter by author disbelief opened on Sep 19, 2023
  10

• Raise error when concurrent to call new Tiktoken

  dqbd/tiktoken#70 · Filter by author qkxie opened on Sep 12, 2023
  2

• Error: null pointer passed to rust

  dqbd/tiktoken#69 · Filter by author emeeks opened on Sep 6, 2023
  4

• NPM package is very huge

  dqbd/tiktoken#68 · Filter by author anzemur opened on Aug 29, 2023
  7

• When I look at cl100k_base.json, I don't see the special tokens like <|im_start|>, but when I add them manually I get an error from the wasm

  dqbd/tiktoken#65 · Filter by author Evertt opened on Aug 5, 2023
  2