Bulk Codebase Tokenizer
Analyze your entire codebase locally. Find duplicate files using SHA-256 and calculate exact LLM token counts (GPT-4, Claude) instantly. Runs 100% in your browser. No uploads.
- No upload
- SHA-256 Hashing
- Token Counting
- 100% Private
Analyze Your Codebase
Drop a `.zip` file or multiple code/text files to find duplicates and calculate total LLM tokens (GPT-4, Claude). 100% local.
- 100% Secure
- No Upload
- Runs in Browser
- Instant Results
- SHA-256 Hashing
Why analyze your codebase?
When feeding your codebase into an LLM (like GPT-4 or Claude) for review or context, every token counts against your context window. Finding and removing duplicate files (like copied configs or redundant components) can save you thousands of tokens and dollars.
Our tool is different. It uses WebAssembly and the Web Crypto API to hash and count your files directly in the browser. No uploads, no server queues, and no privacy risks.
Why this tool is the best choice
Everything runs on your device. Faster, safer and free from the usual online-tool trade-offs.
| Feature | TToolDeck | Typical CLI tools |
|---|---|---|
| No upload to server | Yes | No |
| 100% free with no sign-up | Yes | Partial |
| Runs entirely in your browser | Yes | No |
| No daily usage limits | Yes | No |
| SHA-256 duplicate detection | Yes | Partial |
| Privacy guaranteed | Yes | No |
All developer tools
Free browser-based developer tools. Nothing uploaded, ever.
How it works
- 1
Upload your codebase
Drag and drop a `.zip` file or select individual code files (JS, TS, JSON, etc.).
- 2
Instant analysis
The tool runs SHA-256 hashing and token counting 100% locally in your browser.
- 3
Review the report
See exactly how many tokens you have, which files are duplicates, and what you can safely delete.
Everything you get for free
SHA-256 Duplication
Uses cryptographic hashing to find exact duplicate files in your codebase. Stop paying for the same code twice.
Exact Token Counting
Calculates approximate tokens (1 token ≈ 4 chars) for your entire project. Perfect for estimating GPT-4 or Claude API costs.
ZIP & Folder Support
Drag and drop a `.zip` file or multiple files at once. The tool processes everything locally.
No Setup Required
Forget about installing Python, Node.js, or command-line tools. Works natively in your browser.
Lightning Fast
Uses WebAssembly and Web Crypto API to hash files and count characters at lightning speed.
100% Private
Your codebase never leaves your device. No uploads, no storage, no logs. 100% zero-knowledge.
What people use it for
AI Context Window Management
When feeding a codebase into an LLM, you have a strict token limit. This tool tells you exactly how much space you have left.
Reducing API Costs
Every token costs money. By finding duplicate files and removing them, you can drastically reduce your API bill.
Cleaning Up Legacy Projects
Legacy codebases often have multiple copies of the same utility. Hash-based deduplication helps you find and clean them.
Preparing Training Data
If you are fine-tuning a model, token count and duplicate data can ruin your model. Clean it up before training.
Questions people ask
What is a codebase tokenizer?
It is a tool that counts the number of LLM tokens (like GPT-4 or Claude) in your source code. It is essential for estimating API costs and checking if your codebase fits inside an AI context window.
How does the deduplicator work?
It uses the Web Crypto API to generate a SHA-256 hash for every file. If two files have the exact same hash, they are mathematically identical, meaning one is a duplicate.
Is this tool free?
Yes, it is 100% free with no hidden limits, no daily cap, and no sign-up required.
Does it upload my code to a server?
No. Everything runs in your browser. The files are never sent to any server. This is 100% private and secure.
What file types are supported?
The tool works with `.zip` files and almost all text-based code files: `.js`, `.ts`, `.tsx`, `.jsx`, `.py`, `.json`, `.md`, `.css`, `.html`, and more.
What is a token?
A token is a piece of text that an AI model reads. On average, 1 token is about 4 characters of English text or code.
Does it ignore node_modules?
If you zip your entire project, it will analyze everything inside the zip, including node_modules. We recommend zipping only your source code folder for the best results.
What is the file size limit?
It depends on your device RAM, but ZIP files up to 50MB are processed instantly. Larger files may cause the browser to slow down.
Can I use this for React or Next.js projects?
Yes. It is designed specifically for developers working with modern web frameworks.
Does it support OpenAI tiktoken?
We use a highly accurate approximation (1 token ≈ 4 characters). This is standard for most text and code, though the exact count may vary slightly by model.
Your code never leaves your device
There is no upload step and no account. The entire hashing and token counting process runs inside your browser using WebAssembly and the Web Crypto API. Your code is never sent to a server. Read how we keep it private.