Most search engines rely on stored data — indexed, processed, and ready to query. But I had a different question: What if a search engine could fetch live data and still behave like a distributed system? That question led me to build Disee — a Distributed Search Engine, designed not just to return results, but to simulate real-world distributed system behavior.
Phase 1: The Simple Beginning (Single Node)
Like most projects, this started small:
- A single FastAPI service
- Local text data
- An in-memory inverted index
- A
/searchendpoint
It worked. It was fast. It was clean. But it wasn’t scalable.
The Turning Point
At some point, the real question emerged: How would this system behave under scale? A single node can only do so much—it has limited processing power, no fault tolerance, and no parallelism. That’s when I, along with my teammate Prince, decided to break the system—and rebuild it properly.
Phase 2: Dockerized Multi-Node Aggregation
To eliminate single-node bottlenecks, I introduced a central Gateway Service and split the search orchestration into multiple worker containers running on the same host via Docker Compose.
The gateway sends search queries to all worker nodes in parallel using asynchronous non-blocking HTTP clients (httpx, asyncio), collects worker responses, deduplicates entries, and returns a unified response. This pattern is commonly known as fan-out/fan-in.
Phase 3: Static Multi-Machine Nodes Cluster
Phase 3 takes Disee beyond a single host by distributing execution across physical or virtual machines (currently configured across 3 static worker nodes). This step validates true physical network distribution, handling network latency and remote service discovery.
Dual-Stream Tokenization (Code vs. Prose)
One of the core indexing innovations in Disee is its Dual-Stream Tokenizer. Standard text tokenizers strip punctuation and symbols, turning technical symbols like ctx.execute() into generic ctx and execute tokens. This completely ruins exact code reference matching.
To solve this, Disee concurrently indexes incoming documents into two separate token streams during boot-up:
| Stream | Tokenizer | Behavior | Example |
|---|---|---|---|
| Prose | tokenize_prose() | Extracts clean alpha-numeric terms, discarding common stop words (the, is, at). | "python scripting" → ["python", "scripting"] |
| Code | tokenize_code() | Retains programming syntax characters like underscores, dots, and hyphens intact. | "user_id str.replace" → ["user_id", "str.replace"] |
By retaining precise syntax structures, technical queries hit exact method signatures, variable names, and API paths without interference from general prose terms.
Metadata-Weighted Scoring (Social Boost)
Relevance isn't just about keyword frequency; community validation matters. Disee enriches external documents with metadata (such as StackOverflow upvote counts and edit history) and applies a logarithmic social boost formula:
This logarithmic weighting ensures that high-quality, upvoted answers receive an authority boost without letting popularity completely override query relevance:
- 5 upvotes: ~5% authority boost
- 100 upvotes: ~50% authority boost
- 10,000 upvotes: ~99% authority boost (caps gracefully)
Source-Aware Search Modes
Depending on what the user is looking for, search demands different data sources and tokenization streams. Disee exposes three search modes:
| Mode | External APIs Targeted | Local Index Used |
|---|---|---|
| All (Default) | Wikipedia + StackOverflow | Prose Index |
| Wikipedia | Wikipedia API Only | Prose Index |
| StackOverflow | StackOverflow (StackExchange API) Only | Code Index |
The Real Twist: No Stored Data
Instead of relying on pre-indexed static datasets, Disee fetches real-time data from external APIs (Wikipedia & StackOverflow) on the fly.
Here is how the live distributed pipeline works:
- Query Reception: Gateway receives a live search query.
- Live Retrieval: Gateway fetches real-time data from external APIs.
- Chunking: Gateway splits retrieved content into discrete data chunks.
- Work Distribution: Gateway distributes data chunks asynchronously across worker nodes.
- Parallel Processing: Worker nodes process, attribute, and score data chunks in parallel.
- Aggregation & Fan-In: Gateway aggregates, deduplicates, and ranks worker results into a final output payload.
This architecture shifts the search engine from a static storage engine into a dynamic, parallel stream-processing pipeline.
System Architecture & Startup Auto-Indexing
Disee enforces a strict Gateway-Worker model:
- Gateway: Serves as the entry point for all queries. It fetches external data, distributes workload, and aggregates results.
- Worker Nodes: Receive chunks of data, process content independently, and return structured results.
- Startup Auto-Indexing (
build_index()): On boot-up, each worker node automatically scans internal storage and constructs both code and prose inverted indices in-memory, making the node search-ready instantly.
Interactive Frontend & Visual Identity
A powerful backend deserves a sleek interface. We built a custom React + Tailwind CSS frontend designed with a premium Wedgwood blue and cream color palette:
- Collision Matrix Canvas: A dynamic HTML5 canvas background featuring animated colliding node physics and binary data streams.
- WebM Intro Splash: Seamless video intro animation playing on boot before gracefully cross-fading into the static brand logo.
- Glassmorphic UI: Frosted-glass backdrop filters behind search inputs, filter toggles, and result cards.
Challenges We Faced
Building distributed architectures introduces unique challenges:
1. Asynchronous Complexity
Handling multiple parallel requests wasn’t trivial. Managing timeouts, ensuring async loops don't block the event loop, and debugging asynchronous flow took careful planning.
2. Distributed Debugging
When a query fails, isolating whether the issue lies in the gateway, a specific worker node, or the network layer is difficult. Debugging distributed systems requires centralized tracing.
3. Partial Failures & Graceful Degradation
Not all nodes respond every time. We implemented fallback logic allowing the gateway to return partial results if individual worker nodes time out or fail.
4. Data Consistency vs. Freshness
Since data is fetched live from dynamic APIs, results vary from moment to moment and there is no persistent index. This introduces trade-offs between search freshness and result consistency.
What's Next?
The system is still evolving. Planned roadmap features include:
- Dynamic Node Registration & Heartbeats.
- Fault-tolerant Querying & Circuit Breaking.
- True geographic multi-region cluster deployment.
- Offline indexing pipeline using Apache Spark.
Key Takeaways
- Distributed systems are not about tools — they’re about design decisions.
- Real-time data stream processing introduces complex fault-tolerance trade-offs.
- Async programming is extremely fast — but demands careful timeout and error handling.
- The best way to understand complex systems is to build them from scratch.
"You don’t understand distributed systems by reading about them. You understand them when things break — and you fix them."