The problem: your AI agents don’t have a good place to put what they learn.

RAG pipelines are good at retrieval. They’re not good at accumulation. You ingest documents, chunk them, embed them, and query them — but the pipeline doesn’t build on itself. Every new piece of knowledge has to be shoved into the same undifferentiated vector soup. There’s no structure, no relationships, no concept of “this table depends on that metric.” There’s also no way for agents to write back what they’ve figured out.

OKF is Google’s answer to the question: what if we treated organizational knowledge as a format rather than a platform?

The Core Idea

OKF is simple to state: a bundle is a directory of Markdown files. Each file represents one concept — a database table, a business metric, an API endpoint, a runbook. Files link to each other using standard Markdown links. The whole thing ships as a folder, a tarball, or a git repository.

That’s it. No proprietary database. No vendor SDK. No authentication layer between your agents and the knowledge.

The insight that makes this interesting is the separation between producers and consumers. Whatever writes OKF bundles — an LLM, a script, a human — doesn’t need to know anything about whatever reads them. The format is the contract. A BigQuery enrichment agent can produce a bundle, and a completely unrelated AI assistant can consume it, without either side knowing about the other.

This is what Andrej Karpathy was gesturing at with his “LLM Wiki” idea: instead of repeatedly retrieving from raw documents, have agents incrementally build and maintain a structured wiki. Knowledge compiles once and stays current. OKF formalizes that pattern.

File Format

A concept file looks like this:

---
type: BigQuery Table
title: orders
description: Source of truth for all e-commerce transactions
resource: https://bigquery.googleapis.com/projects/myproject/datasets/ecommerce/tables/orders
tags: [ecommerce, transactions, revenue]
timestamp: 2026-06-12T00:00:00Z
---

## Schema

| Column      | Type      | Description                  |
|-------------|-----------|------------------------------|
| order_id    | INTEGER   | Primary key                  |
| customer_id | INTEGER   | FK → [customers](/tables/customers.md) |
| total_usd   | FLOAT     | Order total before tax       |
| created_at  | TIMESTAMP | UTC creation time            |

## Purpose

This table feeds the [Revenue metric](/metrics/revenue.md) and is the primary
source for the daily orders dashboard.

## Gotchas

`created_at` is stored in UTC but the reporting layer converts to PST.
Don't join against `sessions` on this column directly — use the
`event_date` partition key instead.

The only required frontmatter field is type. Everything else — title, description, resource, tags, timestamp — is optional. The Markdown body is completely free-form. You write what’s useful.

Two filenames are reserved:

• index.md — a bundle overview / table of contents for progressive disclosure
• log.md — a chronological change history in ISO 8601 format

Everything else is up to you.

Bundle Structure

A typical bundle for a data warehouse might look like:

knowledge/
├── index.md
├── log.md
├── tables/
│   ├── orders.md
│   ├── customers.md
│   └── sessions.md
├── metrics/
│   ├── revenue.md
│   ├── dau.md
│   └── conversion_rate.md
└── playbooks/
    ├── incident-response.md
    └── oncall-runbook.md

The cross-references between files (/tables/customers.md, /metrics/revenue.md) create a knowledge graph that both humans and agents can traverse. Navigate it in a text editor, in a git browser, or in one of the reference visualizers Google shipped alongside the spec.

What makes this git-native is that bundles are just directories. You get diffs, blame, branch-based experimentation, and PRs for free. An agent updates metrics/revenue.md and the change shows up in the PR review like any other file edit.

Concept Types

OKF is deliberately unopinionated about what types are valid. The spec defines a few illustrative examples:

You define your own taxonomy. A team building internal tooling might have type: Slack Workflow or type: Terraform Module. The spec doesn’t care.

The type field is the only required field because it’s the minimum needed for tooling to route and categorize concepts — without it, you have no way to tell a metric file from a runbook file programmatically.

Reference Tools

Google shipped three tools alongside the v0.1 spec:

BigQuery enrichment agent — walks your dataset and auto-generates an OKF bundle from table and view definitions. You point it at a BigQuery project and it drafts concept docs for every table, pulling schema information, partition keys, and row counts into structured files.

Static HTML visualizer — renders a bundle as an interactive graph. Single self-contained HTML file, no backend required. You can ship the visualizer alongside a bundle in the same tarball and it works offline.

Sample bundles — reference implementations for GA4 e-commerce, Stack Overflow data, and Bitcoin transaction data. Useful for seeing what a well-structured bundle looks like before you build your own.

These are reference implementations, not the product. The point is to demonstrate that the format is useful without requiring anyone to adopt Google’s specific toolchain.

Use Cases

Self-maintaining data dictionaries. The current state of most data team documentation: stale Confluence pages that were last touched eighteen months ago. An enrichment agent that runs on a schedule, diffs the current schema against the OKF bundle, and opens a PR with updates is genuinely useful. The knowledge doesn’t live in the agent’s context — it lives in files that persist across runs.

Runbooks that agents can read and write. An on-call agent works through an incident response playbook, executes the steps, and appends a ## 2026-06-12 section to the playbook with what it did and what it found. The next time there’s a similar incident, the playbook is richer. This is accumulation, not just retrieval.

Portable client knowledge. If you’re a consultancy or a vendor, you can ship an OKF bundle alongside your automation that documents what the automation does, what data it touches, and how to debug it. The documentation travels with the code in the repo. When the engagement ends, the client has both.

Compliance audit trails. The optional log.md is ISO 8601 timestamped by convention. If you’re writing entries into it as knowledge changes, you get an audit trail of what was known and when without any extra infrastructure. Whether that’s sufficient for actual compliance requirements depends on your context, but it’s a useful starting point.

Agent context bootstrapping. Instead of shoving your entire knowledge base into a system prompt, you give an agent a bundle path and it reads the specific files it needs. The index.md provides navigation. The agent traverses relationships to build up just the context it needs for the task at hand.

How It Compares

It’s worth being clear about what OKF is not, because there are a lot of overlapping things in this space.

OKF vs. RAG. RAG retrieves chunks from a document corpus. OKF stores structured, curated, linked knowledge. They’re complementary — you might RAG over OKF bundles, but OKF bundles are also useful without any RAG in the pipeline. The difference is authorship: RAG sources are usually documents written for humans; OKF files are often written or maintained by agents specifically for agent consumption.

OKF vs. MCP. Model Context Protocol connects agents to tools and data sources at runtime. OKF is static knowledge at rest. An MCP server might expose an OKF bundle as a resource — that’s a sensible combination — but they’re solving different problems.

OKF vs. llms.txt. The llms.txt proposal is a single file convention for mapping a website’s structure. OKF is a multi-file format for representing an organization’s internal knowledge graph. Different scale, different purpose.

OKF vs. Notion/Confluence. These tools have richer editors, mature ecosystems, and better accessibility for non-technical users. OKF’s advantages are: no authentication needed for agents, git-native versioning, zero vendor lock-in, and a format that doesn’t change on you. If your team already lives in Notion, OKF doesn’t replace it — but you could export a Notion database as an OKF bundle and the agent-facing properties become much simpler.

OKF vs. data catalogs. Tools like Datahub or Alation are heavyweight: they require dedicated infrastructure, have their own data models, and are mostly read-only for agents. OKF is lightweight enough that a shell script can produce a valid bundle. Whether that simplicity is a feature or a limitation depends on your scale.

What’s Missing (So Far)

v0.1 means incomplete by design. A few things are notably absent:

Access control. A bundle is a directory. If an agent can read the directory, it can read everything. For sensitive knowledge (PII policies, security runbooks), you’d need to handle this at the filesystem or repository level, outside the spec.

Conflict resolution. If two agents write to the same file concurrently, you have a merge conflict like any other. The spec doesn’t define a strategy for this. In practice, you’d probably handle it the same way you handle concurrent git edits — with locks, queuing, or PR-based workflows.

Query semantics. There’s no defined way to query across a bundle beyond “read the files you need.” For small bundles this is fine. For a bundle with thousands of concept files, you’d want some kind of index — either an LLM-navigable index.md hierarchy or a simple key-value lookup by type and name. The spec doesn’t specify this.

Validation beyond type. The only required field is type. There’s a validator for v0.1 compliance, but nothing prevents a concept file from having incorrect schema information or outdated descriptions. Data quality is still a human (or agent) responsibility.

Why This Matters Now

The timing makes sense. The last year has produced solid tooling for agents that act (MCP, tool use, function calling) but relatively little for agents that know and remember. Context windows are larger but still finite. RAG is mature but doesn’t accumulate. Fine-tuning is expensive and static.

A simple, portable, text-based format for persistent structured knowledge fills a gap that wasn’t filled by any of the above. The fact that it’s just Markdown in a folder means the barrier to adoption is nearly zero — if your agents can write files, they can write OKF.

Whether OKF specifically becomes a standard or whether it gets superseded by something better, the pattern it embodies — agents maintaining interlinked knowledge files — is going to be a fixture of how AI systems work. The question of where knowledge lives, who maintains it, and how it’s structured is one of the genuinely interesting open problems in applied AI right now.

OKF is a reasonable opening move.

I’ll walk through building a document Q&A service end to end — ingestion, retrieval, FastAPI serving, and the parts that bit me along the way.

What We’re Building

A service that:

1. Accepts document uploads (PDF, text, markdown)
2. Chunks and embeds them into a vector store
3. Exposes an API endpoint where you can ask questions about uploaded documents
4. Returns answers with source citations

The architecture is straightforward:

Client → FastAPI → RAG Pipeline → LLM
                       ↑
                 Vector Store (Qdrant)
                       ↑
              Document Ingestion Worker

The stack: FastAPI for the API, Qdrant for the vector store (running locally via Docker), LangChain for orchestration, OpenAI for embeddings and generation.

Project Structure

doc-qa-service/
├── app/
│   ├── __init__.py
│   ├── main.py          # FastAPI application
│   ├── ingest.py        # Document ingestion pipeline
│   ├── retrieval.py     # Query + generation
│   ├── config.py        # Settings
│   └── models.py        # Pydantic models
├── docker-compose.yml   # Qdrant + service
├── Dockerfile
├── requirements.txt
└── README.md

Step 1: Configuration

Using Pydantic’s BaseSettings to manage config — this makes switching between dev and prod environments clean.

# app/config.py
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    openai_api_key: str
    qdrant_host: str = "localhost"
    qdrant_port: int = 6333
    collection_name: str = "documents"
    embedding_model: str = "text-embedding-3-small"
    llm_model: str = "gpt-4o-mini"
    chunk_size: int = 512
    chunk_overlap: int = 64
    retrieval_k: int = 4

    class Config:
        env_file = ".env"

settings = Settings()

Step 2: Document Ingestion

The ingestion module handles loading, chunking, and storing documents. I wanted this to be callable both directly and via the API endpoint.

# app/ingest.py
import hashlib
import uuid
from pathlib import Path
from typing import Optional

from langchain_community.document_loaders import (
    PyPDFLoader,
    TextLoader,
    UnstructuredMarkdownLoader,
)
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams

from .config import settings


def get_loader(file_path: str):
    ext = Path(file_path).suffix.lower()
    loaders = {
        ".pdf": PyPDFLoader,
        ".txt": TextLoader,
        ".md": UnstructuredMarkdownLoader,
    }
    loader_class = loaders.get(ext)
    if not loader_class:
        raise ValueError(f"Unsupported file type: {ext}")
    return loader_class(file_path)


def ensure_collection(client: QdrantClient, dimension: int = 1536):
    """Create collection if it doesn't exist."""
    existing = [c.name for c in client.get_collections().collections]
    if settings.collection_name not in existing:
        client.create_collection(
            collection_name=settings.collection_name,
            vectors_config=VectorParams(size=dimension, distance=Distance.COSINE),
        )


def ingest_document(file_path: str, doc_id: Optional[str] = None) -> dict:
    """
    Load, chunk, embed, and store a document.
    Returns metadata about the ingestion.
    """
    if doc_id is None:
        # Deterministic ID based on file content
        content_hash = hashlib.sha256(
            Path(file_path).read_bytes()
        ).hexdigest()[:16]
        doc_id = content_hash

    loader = get_loader(file_path)
    docs = loader.load()

    splitter = RecursiveCharacterTextSplitter(
        chunk_size=settings.chunk_size,
        chunk_overlap=settings.chunk_overlap,
    )
    chunks = splitter.split_documents(docs)

    # Tag each chunk with the doc_id for later filtering
    for chunk in chunks:
        chunk.metadata["doc_id"] = doc_id
        chunk.metadata["source_file"] = Path(file_path).name

    embeddings = OpenAIEmbeddings(
        model=settings.embedding_model,
        api_key=settings.openai_api_key,
    )

    client = QdrantClient(host=settings.qdrant_host, port=settings.qdrant_port)
    ensure_collection(client)

    vectorstore = QdrantVectorStore(
        client=client,
        collection_name=settings.collection_name,
        embedding=embeddings,
    )
    vectorstore.add_documents(chunks)

    return {
        "doc_id": doc_id,
        "chunks_stored": len(chunks),
        "source_file": Path(file_path).name,
    }

A few decisions worth noting:

• The deterministic doc_id means re-ingesting the same file is idempotent in terms of tracking, though you’d want deduplication logic in the vector store for production use
• Tagging chunks with doc_id allows scoped queries later (“only search within this document”)

Step 3: Retrieval and Generation

# app/retrieval.py
from typing import Optional
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue

from .config import settings


SYSTEM_PROMPT = """You are a helpful assistant that answers questions based on provided document context.

Rules:
- Answer only based on the context provided
- If the context doesn't contain the answer, say "I don't have enough information in the provided documents to answer this"
- Cite the source document and page number when available
- Be concise but complete

Context:
{context}

Question: {question}

Answer:"""


def get_vectorstore(doc_id: Optional[str] = None) -> QdrantVectorStore:
    client = QdrantClient(host=settings.qdrant_host, port=settings.qdrant_port)
    embeddings = OpenAIEmbeddings(
        model=settings.embedding_model,
        api_key=settings.openai_api_key,
    )

    vectorstore = QdrantVectorStore(
        client=client,
        collection_name=settings.collection_name,
        embedding=embeddings,
    )
    return vectorstore


def answer_question(question: str, doc_id: Optional[str] = None) -> dict:
    vectorstore = get_vectorstore()

    retriever_kwargs = {"k": settings.retrieval_k}

    # Optionally scope to a specific document
    if doc_id:
        retriever_kwargs["filter"] = Filter(
            must=[
                FieldCondition(
                    key="metadata.doc_id",
                    match=MatchValue(value=doc_id),
                )
            ]
        )

    retriever = vectorstore.as_retriever(search_kwargs=retriever_kwargs)

    prompt = PromptTemplate(
        template=SYSTEM_PROMPT,
        input_variables=["context", "question"],
    )

    llm = ChatOpenAI(
        model=settings.llm_model,
        temperature=0,
        api_key=settings.openai_api_key,
    )

    qa_chain = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=retriever,
        return_source_documents=True,
        chain_type_kwargs={"prompt": prompt},
    )

    result = qa_chain.invoke({"query": question})

    # Extract source citations
    sources = []
    for doc in result.get("source_documents", []):
        sources.append({
            "file": doc.metadata.get("source_file", "unknown"),
            "page": doc.metadata.get("page", None),
            "doc_id": doc.metadata.get("doc_id"),
        })

    # Deduplicate sources
    seen = set()
    unique_sources = []
    for s in sources:
        key = (s["file"], s["page"])
        if key not in seen:
            seen.add(key)
            unique_sources.append(s)

    return {
        "answer": result["result"],
        "sources": unique_sources,
    }

Step 4: FastAPI Application

# app/main.py
import shutil
import tempfile
from pathlib import Path

from fastapi import FastAPI, File, HTTPException, UploadFile
from fastapi.middleware.cors import CORSMiddleware

from .ingest import ingest_document
from .models import QuestionRequest, QuestionResponse, IngestResponse
from .retrieval import answer_question

app = FastAPI(
    title="Document Q&A Service",
    description="Upload documents and ask questions about them",
    version="0.1.0",
)

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)


@app.get("/health")
def health():
    return {"status": "ok"}


@app.post("/ingest", response_model=IngestResponse)
async def ingest(file: UploadFile = File(...)):
    """Upload and ingest a document (PDF, TXT, or MD)."""
    allowed = {".pdf", ".txt", ".md"}
    ext = Path(file.filename).suffix.lower()
    if ext not in allowed:
        raise HTTPException(
            status_code=400,
            detail=f"Unsupported file type '{ext}'. Allowed: {allowed}"
        )

    # Write to temp file — the loaders need a file path
    with tempfile.NamedTemporaryFile(suffix=ext, delete=False) as tmp:
        shutil.copyfileobj(file.file, tmp)
        tmp_path = tmp.name

    try:
        result = ingest_document(tmp_path)
    finally:
        Path(tmp_path).unlink(missing_ok=True)

    return IngestResponse(**result)


@app.post("/ask", response_model=QuestionResponse)
async def ask(request: QuestionRequest):
    """Ask a question about ingested documents."""
    if not request.question.strip():
        raise HTTPException(status_code=400, detail="Question cannot be empty")

    result = answer_question(
        question=request.question,
        doc_id=request.doc_id,
    )
    return QuestionResponse(**result)

# app/models.py
from typing import List, Optional
from pydantic import BaseModel


class IngestResponse(BaseModel):
    doc_id: str
    chunks_stored: int
    source_file: str


class QuestionRequest(BaseModel):
    question: str
    doc_id: Optional[str] = None  # Scope to a specific document


class Source(BaseModel):
    file: str
    page: Optional[int] = None
    doc_id: Optional[str] = None


class QuestionResponse(BaseModel):
    answer: str
    sources: List[Source]

Step 5: Running It

docker-compose.yml:

services:
  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "6333:6333"
      - "6334:6334"
    volumes:
      - qdrant_storage:/qdrant/storage

  api:
    build: .
    ports:
      - "8000:8000"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - QDRANT_HOST=qdrant
    depends_on:
      - qdrant

volumes:
  qdrant_storage:

Dockerfile:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

requirements.txt:

fastapi==0.109.0
uvicorn[standard]==0.27.0
langchain==0.1.0
langchain-openai==0.0.5
langchain-qdrant==0.0.1
langchain-community==0.0.15
qdrant-client==1.7.0
pypdf==4.0.1
unstructured==0.12.0
pydantic-settings==2.1.0
python-multipart==0.0.6

Start everything with:

export OPENAI_API_KEY="your-key"
docker-compose up

Testing the Service

# Upload a document
curl -X POST "http://localhost:8000/ingest" \
  -F "file=@/path/to/your/document.pdf"

# Response:
# {"doc_id": "a3f2b1c4", "chunks_stored": 47, "source_file": "document.pdf"}

# Ask a question
curl -X POST "http://localhost:8000/ask" \
  -H "Content-Type: application/json" \
  -d '{"question": "What are the main conclusions?", "doc_id": "a3f2b1c4"}'

# Response:
# {
#   "answer": "The main conclusions are...",
#   "sources": [{"file": "document.pdf", "page": 12, "doc_id": "a3f2b1c4"}]
# }

Interactive API docs at http://localhost:8000/docs (FastAPI’s built-in Swagger UI).

Lessons from Making This Production-Ready

Mistake 1: Not handling Qdrant connection failures gracefully. The Qdrant client will throw on connection refused. Wrap your vectorstore operations in try/except and return appropriate HTTP status codes.

Mistake 2: Blocking the event loop during ingestion. ingest_document is CPU and I/O heavy. For production, move it to a background task or a queue (Celery, ARQ, etc.) and return a job ID immediately.

@app.post("/ingest")
async def ingest(background_tasks: BackgroundTasks, file: UploadFile = File(...)):
    # ... save file ...
    job_id = str(uuid.uuid4())
    background_tasks.add_task(ingest_document, tmp_path, job_id)
    return {"job_id": job_id, "status": "processing"}

Mistake 3: Not deduplicating on re-upload. If someone uploads the same document twice, you get duplicate chunks in the vector store which inflate your results. Use the content hash as doc_id and check if it already exists before re-ingesting.

Mistake 4: Trusting the LLM to cite correctly. The LLM will sometimes fabricate citations. Don’t ask the LLM to produce source references — collect them from the retrieved chunks yourself (as shown in the retrieval.py above) and append them to the response separately.

The service above is functional but the path from here to something truly production-ready involves proper auth, rate limiting, background job processing, observability, and evaluation. I’ll cover those in subsequent posts.

What is Snowflake ID?

Snowflake ID is a 64-bit unique identifier originally developed by Twitter. It is highly efficient and scalable, designed to work in distributed systems. The ID consists of the following components:

1. Timestamp (41 bits): Represents the time in milliseconds since a custom epoch.
2. Machine ID (10 bits): Uniquely identifies the machine or node generating the ID.
3. Sequence Number (12 bits): Ensures uniqueness for IDs generated within the same millisecond.

Why Use Snowflake ID?

1. Scalability: Works seamlessly in distributed environments.
2. Uniqueness: Combines timestamp, machine ID, and sequence number to ensure no duplicates.
3. Efficiency: Generates IDs in constant time, even under high loads.
4. Compactness: Represents large IDs in a compact 64-bit format.

Snowflake ID Python Implementation

Below is a Python implementation of a Snowflake ID generator and its integration with MongoDB:

import os
import time
from pymongo import MongoClient
from pymongo.errors import DuplicateKeyError

class SnowflakeIDGenerator:
    """
    A class to generate unique Snowflake IDs.

    Attributes:
        epoch (int): The custom epoch timestamp in milliseconds. Default is 1640995200000.
        machine_id (int): The machine ID, derived from the environment variable "MACHINE_ID" and masked with 0x3FF.
        sequence (int): The sequence number for IDs generated within the same millisecond.
        last_timestamp (int): The timestamp of the last generated ID.

    Methods:
        _current_timestamp(): Returns the current timestamp in milliseconds.
        generate_id(): Generates a unique Snowflake ID based on the current timestamp, machine ID, and sequence number.
    """
    def __init__(self, epoch: int = 1640995200000):
        self.machine_id = int(os.getenv("MACHINE_ID", "0")) & 0x3FF
        self.epoch = epoch
        self.sequence = 0
        self.last_timestamp = -1

    def _current_timestamp(self):
        return int(time.time() * 1000)

    def generate_id(self):
        timestamp = self._current_timestamp()

        if timestamp == self.last_timestamp:
            self.sequence = (self.sequence + 1) & 0xFFF
            if self.sequence == 0:
                while timestamp <= self.last_timestamp:
                    timestamp = self._current_timestamp()
        else:
            self.sequence = 0

        self.last_timestamp = timestamp

        return (
            ((timestamp - self.epoch) << 22) |
            (self.machine_id << 12) |
            self.sequence
        )

# MongoDB connection
client = MongoClient(os.getenv("MONGO_URI", "mongodb://mongo:27017/"))
db = client["test_database"]
collection = db["test_collection"]

# Initialize Snowflake Generator
generator = SnowflakeIDGenerator()

# Generate and insert IDs
batch_size = int(os.getenv("BATCH_SIZE", "100000"))
for i in range(batch_size):
    unique_id = generator.generate_id()
    document = {
        "_id": unique_id,
        "pod": os.getenv("HOSTNAME", "unknown"),
        "timestamp": time.time()
    }
    try:
        collection.insert_one(document)
    except DuplicateKeyError:
        print(f"Document with _id {document['_id']} already exists.")

print(f"Generated and inserted {batch_size} IDs.")

This application generates unique Snowflake IDs and writes them to a MongoDB database. Each ID is stored in a document along with metadata, such as the hostname and timestamp, enabling scalable storage and retrieval in distributed systems.

Deploying the Application Using Docker

To simplify deployment, we use Docker and Docker Compose. Below is the Dockerfile and docker-compose.yml configuration:

Dockerfile

# Use an official Python runtime as the base image
FROM python:3.10-slim

# Set the working directory in the container
WORKDIR /app

# Copy application code
COPY app.py /app

# Install dependencies
RUN pip install pymongo

# Define environment variables
ENV MONGO_URI=mongodb://mongo:27017/
ENV MACHINE_ID=0

# Run the application
CMD ["python", "app.py"]

Docker Compose

services:
  mongo:
    image: mongo:latest
    container_name: mongo
    ports:
      - "27017:27017"
    volumes:
      - mongo_data:/data/db

  app1:
    image: snowflakeidgen
    build:
      context: .
      dockerfile: Dockerfile
    environment:
      - MONGO_URI=mongodb://mongo:27017/
      - MACHINE_ID=1
    depends_on:
      - mongo

  app2:
    image: snowflakeidgen
    environment:
      - MONGO_URI=mongodb://mongo:27017/
      - MACHINE_ID=2
    depends_on:
      - mongo

  app3:
    image: snowflakeidgen
    environment:
      - MONGO_URI=mongodb://mongo:27017/
      - MACHINE_ID=3
    depends_on:
      - mongo

volumes:
  mongo_data:

Running the Application

1. Build the Docker Image:

   docker-compose build
   

2. Start the Services:

   docker-compose up
   

3. Verify the Output:

   • The MongoDB container will be accessible at localhost:27017.
   • The application will generate and insert unique Snowflake IDs into MongoDB.

Short version: it’s meaningfully better for prototyping and small workloads. For production at scale, the tradeoffs are more nuanced.

Background: How Pinecone Used to Work

Before serverless, Pinecone used a pod-based model. You provisioned dedicated pods (p1, p2, s1) and paid by the hour regardless of whether you were querying or not. A pod running 24/7 costs money whether you have 100 queries a day or 100,000.

This made sense for production workloads with predictable traffic. It made less sense if you were:

• Building a prototype that gets used a few times a week
• Running experiments across multiple indexes
• Doing batch jobs that query heavily for an hour then go quiet

The minimum viable setup (1 x s1.x1 pod) ran about $70/month. Not huge, but enough to make you think twice about spinning up indexes for exploration.

What Serverless Actually Is

Serverless Pinecone decouples storage from compute. Your vectors live in blob storage (AWS S3 under the hood). When a query comes in, Pinecone spins up compute to search across that storage, then bills you for the query itself.

Billing shifts from time-based to usage-based:

• Storage: ~$0.033/GB/month
• Reads: ~$8 per million read units (each query uses multiple read units depending on your index size)
• Writes: ~$2 per million write units

The free tier is genuinely useful now — 2GB of storage and a monthly allowance of read/write units. That’s enough for real experimentation.

Setting Up a Serverless Index

The code change is small. The key difference is in the spec parameter when creating an index:

Before (pod-based):

from pinecone import Pinecone, PodSpec

pc = Pinecone(api_key="your-api-key")

pc.create_index(
    name="my-index",
    dimension=1536,
    metric="cosine",
    spec=PodSpec(
        environment="gcp-starter",
        pod_type="p1.x1",
        pods=1
    )
)

After (serverless):

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="your-api-key")

pc.create_index(
    name="my-index",
    dimension=1536,
    metric="cosine",
    spec=ServerlessSpec(
        cloud="aws",
        region="us-east-1"
    )
)

That’s it. The index operations (upsert, query, delete) are identical.

Before vs After: A Real Comparison

I ran the same workload — a small document Q&A system with ~50K vectors — against both setups. Here’s what I observed:

Setup Experience

Pod-based: Creating an index took 2–5 minutes while the pod initialized. You’d see a “Initializing” state in the dashboard. Frustrating when you just want to test something quickly.

Serverless: Index creation is near-instant. Available in under 10 seconds in my tests. This sounds minor but it substantially changes the experimentation loop.

Query Latency

This is where serverless has a genuine tradeoff. Cold start behavior is noticeable:

Pod-based (p1.x1, warm):   8–15ms average query latency
Serverless (warm):         15–30ms average query latency
Serverless (cold start):   200–800ms first query after idle period

For a RAG application where you’re chaining LLM calls anyway, the extra 15ms on warm queries doesn’t matter. The cold start does matter if your application has spiky traffic.

Pinecone has improved this significantly over 2024 — cold starts are faster than when serverless first launched.

Cost at Different Scales

I ran some rough estimates for a document Q&A workload:

The crossover point is somewhere around 10–15K queries/day depending on your index size and query complexity. Below that, serverless wins on cost. Above that, it depends heavily on your traffic pattern.

Migrating an Existing Index

There’s no in-place migration — you create a new serverless index and reindex your data. With LangChain this is straightforward:

from pinecone import Pinecone, ServerlessSpec
from langchain_pinecone import PineconeVectorStore
from langchain_openai import OpenAIEmbeddings

pc = Pinecone(api_key="your-api-key")

# Create new serverless index
pc.create_index(
    name="my-index-serverless",
    dimension=1536,
    metric="cosine",
    spec=ServerlessSpec(cloud="aws", region="us-east-1")
)

# Re-embed and upsert your documents
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = PineconeVectorStore.from_documents(
    documents=your_chunks,
    embedding=embeddings,
    index_name="my-index-serverless"
)

If your source documents are still available (they should be — don’t rely on the vector store as your source of truth), this is a straightforward operation. The re-embedding cost is real though — factor that in if you have millions of vectors.

What I’d Use Now

For new projects in late 2024, my default is serverless unless I have a concrete reason for the pod-based model. The reasons to stick with pods:

1. Consistent sub-20ms latency with no cold starts
2. You’re already paying for the pod and don’t want to migrate
3. You need features that are pod-only (metadata filtering at very large scale behaves differently)

For everything else — prototypes, low-to-medium traffic production apps, experiments — serverless is the better default. The free tier is enough to build real things, and the pay-per-use model means you’re not bleeding money on idle resources.

The user experience around indexing being fast now is actually the biggest win in day-to-day use. Faster feedback loop matters more than I expected.

Quick Reference

# Full working example - serverless index with LangChain
import os
from pinecone import Pinecone, ServerlessSpec
from langchain_pinecone import PineconeVectorStore
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA

pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])

index_name = "rag-demo"

# Create if it doesn't exist
if index_name not in pc.list_indexes().names():
    pc.create_index(
        name=index_name,
        dimension=1536,
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-east-1")
    )

# Use with LangChain
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = PineconeVectorStore(
    index_name=index_name,
    embedding=embeddings
)

retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

qa = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever
)

result = qa.invoke({"query": "Your question here"})
print(result["result"])

pip install pinecone langchain langchain-pinecone langchain-openai

I want to use this as a foundation post for a series I’m planning on AI Pipelines and applied LLM work. A lot of what I’ll write going forward builds on these ideas, so it’s worth getting the basics right.

Why RAG Exists

Large language models have a knowledge cutoff. Ask GPT-4 about something that happened last month and it either hallucinates or tells you it doesn’t know. More practically, ask it about your company’s internal documentation and it definitely doesn’t know.

The naive fix is fine-tuning — train the model on your data. The problems: fine-tuning is expensive, slow, and static. Every time your data changes you need to re-train.

Retrieval-Augmented Generation (RAG) is the other approach. Instead of baking knowledge into the model weights, you retrieve relevant context at query time and feed it to the model as part of the prompt. The model’s job becomes: “given this context, answer this question.”

This is powerful because:

• Your knowledge base can be updated without touching the model
• You can trace where the answer came from (the retrieved chunks)
• It works with any LLM — even smaller, cheaper models improve dramatically with good context

The Five Components

A RAG pipeline has five logical pieces. Let me walk through each one.

1. Document Ingestion

Before you can retrieve anything, you need to get your documents into a format the pipeline can use. This typically means:

• Loading: Reading PDFs, web pages, markdown files, database exports, etc.
• Chunking: Splitting long documents into smaller pieces
• Metadata extraction: Keeping track of source, date, author

The chunking step is more art than science. Too small and individual chunks lose context. Too large and you’re paying for tokens you don’t need and diluting relevance.

A simple recursive character splitter is a reasonable starting point:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = splitter.split_documents(docs)

The chunk_overlap matters — it ensures that ideas that span chunk boundaries aren’t silently cut off.

2. Embedding

Each chunk gets converted into a dense vector — a list of floating point numbers that represents the semantic meaning of the text. Semantically similar text ends up close together in this vector space.

The embedding model does this translation. Common choices in late 2024:

• OpenAI text-embedding-3-small — cheap, good quality, 1536 dimensions
• OpenAI text-embedding-3-large — higher quality, 3072 dimensions
• sentence-transformers/all-MiniLM-L6-v2 — local, fast, smaller (384 dims)
• Cohere embed-v3 — strong multilingual support

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Embed a single string
vector = embeddings.embed_query("What is a transformer architecture?")
print(len(vector))  # 1536

3. Vector Store

The vectors need to live somewhere searchable. A vector store indexes embeddings so you can find the nearest neighbors to a query vector quickly — usually using HNSW (Hierarchical Navigable Small World) or FAISS under the hood.

Options range from in-memory (good for prototyping) to fully managed cloud databases:

For a quick local setup with ChromaDB:

from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
    persist_directory="./chroma_db"
)

4. Retrieval

At query time, the user’s question gets embedded using the same model, and you find the k most similar chunks in the vector store.

query = "How does attention mechanism work in transformers?"
results = vectorstore.similarity_search(query, k=4)

for doc in results:
    print(doc.metadata)
    print(doc.page_content[:200])
    print("---")

This is where a lot of the tuning happens in practice. Raw cosine similarity isn’t always what you want. You might want:

• MMR (Maximal Marginal Relevance) — balance relevance with diversity so you don’t retrieve 4 chunks that all say the same thing
• Hybrid search — combine dense (semantic) with sparse (keyword/BM25) retrieval
• Reranking — use a second model to re-score the retrieved chunks

5. Generation

The retrieved chunks get assembled into a prompt along with the user’s question, and sent to an LLM:

from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

template = """Use the following context to answer the question.
If you don't know the answer based on the context, say so — don't make things up.

Context:
{context}

Question: {question}

Answer:"""

prompt = PromptTemplate(
    input_variables=["context", "question"],
    template=template
)

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    chain_type_kwargs={"prompt": prompt}
)

response = qa_chain.invoke({"query": "How does attention work?"})
print(response["result"])

Putting It All Together

Here’s a minimal end-to-end pipeline — load some documents, embed them, store them, and query:

import os
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

# 1. Load documents
loader = DirectoryLoader("./docs", glob="**/*.md", loader_cls=TextLoader)
docs = loader.load()
print(f"Loaded {len(docs)} documents")

# 2. Chunk
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_documents(docs)
print(f"Split into {len(chunks)} chunks")

# 3. Embed + store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# 4. Build retrieval chain
template = """Answer based on the context below. Say "I don't know" if the context
doesn't contain the answer.

Context: {context}
Question: {question}
Answer:"""

qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4o-mini", temperature=0),
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    chain_type_kwargs={
        "prompt": PromptTemplate(
            template=template,
            input_variables=["context", "question"]
        )
    }
)

# 5. Query
while True:
    question = input("\nAsk a question (or 'quit'): ")
    if question.lower() == "quit":
        break
    result = qa.invoke({"query": question})
    print(f"\n{result['result']}")

What This Doesn’t Cover (Yet)

This is the happy path. Production RAG introduces a lot of complexity I want to address in separate posts:

• Evaluation — how do you know if your RAG is actually good? RAGAS, LlamaIndex eval frameworks
• Chunking strategies — semantic chunking, parent-child chunking, document hierarchy
• Hybrid search — combining dense and sparse retrieval
• Reranking — Cohere Rerank, cross-encoder models
• Agentic retrieval — query decomposition, iterative retrieval

The pipeline I’ve shown here is functional but naive. It works well enough to prototype on. The gap between this prototype and a production system is where most of the interesting engineering lives, and that’s what I’ll be exploring in upcoming posts.

Prerequisites

If you want to run the code above:

pip install langchain langchain-openai langchain-chroma chromadb
export OPENAI_API_KEY="your-key-here"

Put some markdown or text files in a ./docs directory and you’re off.

Twitter is one of the main sources of media consumption. So why not bring my notes into my twitter feed periodically so I get to look at them more often? That gave birth to a weekend project - Making a twitter bot to read my notes and tweet a note based on my interests. I tagged each note as I took them, so it was easier to choose the ones that mattered to me at that time. It also gave me a choice to tune my bot in such a way that certain notes get tweeted with higher probability. After some fun time with Google Cloud Run in the past, I was reading about Cloud Functions. It fitted perfectly into the use case I was working on. I can have the bot run in a schedule through Cloud Scheduler. The number of invocations, CPU time and network egress for my use case were very minimal and fell well under the generous free limits. Putting all together, I came up with something like below.

A little bit about the individual components:

• NotesBot
  • An event driven Cloud Function in Python, to post a note into one or more tweets
  • Gets a random note from the Note Service
  • Processes the note and post them using Tweepy and Twitter v2 API
  • Set to be invoked when a message is posted to the connected pub-sub
  • Tweets are posted @deeptechnotes handle
• A Cloud Scheduler job to post a message to the pub-sub on a cadence (e.g. every 3 hours)
• Note Service
  • A HTTP Cloud function in Go to serve a note from my collection. This could have been done with Python/Flask too, went with Go just for some fun as I was learning Go in the recent times. To keep things simple, didn’t add any API spec when I started off. Planning to add that soon.
  • Supports different paths to return a random note or multiple notes based on tags specified in the query
  • Having this as a separate service makes it convenient to pull the notes from other possible clients (e.g. a vscode extension to display my notes in a notification pop-up, chrome extension) in future
  • The service is available here. Some sample queries:

➜  ~ curl -s "https://mynotesapp-nxxo6p55tq-uc.a.run.app/" | jq                        
{                                                                              
  "id": 9,                                                                     
  "description": "shell - sum numbers from stdin",                                                                                                             
  "tags": [
    "shell",
    "commands",                                                                                                                                                
    "tools"
  ],
  "contents": [
    "# Ways to sum up numbers from stdin or from piped output from previous commands",
    "# Using paste command",
    "$ echo somefile | paste -s -d+ | bc",
    "$ cat nums | paste -sd+ - | bc",
    "# using awk.. substitute $1 with the actual column number",
    "$ cat nums | awk '{sum += $1} END {print sum}'"
  ]
}

➜  ~ curl -s "https://mynotesapp-nxxo6p55tq-uc.a.run.app/notes?tags=vim&limit=1" | jq
[
  {
    "id": 65,
    "description": "vim - show relative and absolute file path",
    "tags": [
      "vim",
      "programming",
      "editor",
      "shortcuts"
    ],
    "contents": [
      "Ctrl+G - shows the relative path of the current file",
      "{n}Ctrl+G - shows the relative path of the nth file in the buffer",
      "1Ctrl+G - shows the absolute path of the current file"
    ]
  }
]

It would be interesting to have an vscode extension too, that displays my notes in a pop-up. May be for another weekend.

• Schema Definition: Protocol Buffers require a schema definition that defines the structure of the message, which is then used to generate code in various languages. On the other hand, JSON does not require a schema definition, and its structure can vary widely depending on the implementation. Makes it super easy to write and test simple cases. No overhead of compiling the schema definition.

• Language Support: Protocol Buffers provide first-class support for multiple languages, including Java, C++, Python, and Go, and support for additional languages can be added through extensions. JSON, on the other hand, has ubiquitous support across all languages.

• Data Types: Protocol Buffers support a smaller set of data types than JSON, including strings, numbers, booleans, enums, and arrays. JSON supports a wider range of data types, including null, objects, and custom data types.

• Parsing Overhead: JSON requires a parser to be used to parse the data, which can have a performance overhead. Protocol Buffers do not require a parser, and the data can be directly deserialized into objects.

• Readability: JSON is more human-readable and easier to understand than Protocol Buffers, making it easier to debug and develop applications that consume JSON data.

Defined a simple protobuf definition with some scalar types.

syntax = "proto3";
option go_package = "github.com/deepns/codegym/go/learning/grpc/echo";

message EchoRequestWithCount {
    string message = 1;
    int32 count = 2;  
}

Using Go as the language choice, compiled the above message with a protoc compiler. It created a struct type, with some unexported fields and with the message fields exported.

type EchoRequestWithCount struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Message string `protobuf:"bytes,1,opt,name=message,proto3" json:"message,omitempty"`
	Count   int32  `protobuf:"varint,2,opt,name=count,proto3" json:"count,omitempty"`
}

Lets see how the encoded type loos like.

import (
    "fmt"
    pb "github.com/deepns/codegym/go/learning/grpc/echo/echo"
    "github.com/golang/protobuf/proto"
)

func main() {
	echoRequest := pb.EchoRequestWithCount{
		Message: "Woof!",
		Count:   100,
	}
	echoRequestBinary, _ := proto.Marshal(&echoRequest)
	fmt.Println("echoRequestBinary:", echoRequestBinary)
}

The output came as:

echoRequestBinary: [10 5 87 111 111 102 33 16 100]

I guess 10 corresponds to byte, followed by length of the string (which is 5 in this case), then type of int32, determined by 16 followed by the actual value. For larger integers, the encoding scheme seems different.

To explore further, defined another message one with more types.

syntax = "proto3";

option go_package = "github.com/deepns/codegym/go/learning/grpc/protobuf/books";

// Book attributes defined with scalar fields
message Book {
    string title = 1;
    uint32 year = 2;
    double price = 3;
    bool is_released = 4;
    BookGenre genre = 5;
}

enum BookGenre {
    FICTION = 0;
    THRILLER = 1;
    MEMOIR = 2;
}

message Shelf { 
    repeated Book books_to_read = 1;
    repeated Book books_read = 2;
}

Then tried to create some objects for the above types and serialize them into a file.

package main

import (
	"encoding/json"
	"log"

	pb "github.com/deepns/codegym/go/learning/grpc/protobuf/books"
	"google.golang.org/protobuf/proto"
)

func main() {
    shelf := &pb.Shelf{
		BooksToRead: []*pb.Book{
			{
				Title:      "Sapiens: A Brief History of Humankind",
				Year:       2015,
				Price:      12.99,
				IsReleased: true,
				Genre:      pb.BookGenre_MEMOIR,
			},
			{
				Title:      "The Water Dancer",
				Year:       2019,
				Price:      11.99,
				IsReleased: true,
				Genre:      pb.BookGenre_FICTION,
			},
		},
		BooksRead: []*pb.Book{
			{
				Title:      "The Underground Railroad",
				Year:       2016,
				Price:      9.99,
				IsReleased: true,
				Genre:      pb.BookGenre_FICTION,
			},
			{
				Title:      "The Power of Now: A Guide to Spiritual Enlightenment",
				Year:       1997,
				Price:      7.99,
				IsReleased: true,
				Genre:      pb.BookGenre_MEMOIR,
			},
		},
	}

	// Marshal the "shelf" instance into binary format
	shelfData, err := proto.Marshal(shelf)
	if err != nil {
		log.Fatalln("Error marshaling shelf data", err)
	}

    // Marshal the "shelf" instance into json format
	// Since json encoding is defined in the struct itself, protobuf struct
	// can be readily marshaled to json.
	shelfDataJson, err := json.Marshal(shelf)
	if err != nil {
		log.Fatalln("Error marshaling shelf data to json", err)
	}

    log.Println("Bytes written in pb format:", len(shelfData))
	log.Println("Bytes written in json format:", len(shelfDataJson))

2023/03/30 18:30:38 Bytes written in pb format: 205
2023/03/30 18:30:38 Bytes written in json format: 413

Protobuf bytes took about 205 bytes whereas the json based data bytes took about 413 bytes in this case, ~50% reduction. This is of course highly simplified example, as this could vary depending on the length of the keys and values. These savings on data size and transfer do provide significant values in high performance applications.

The library provides a set of functions for creating and managing ZooKeeper sessions, creating and deleting nodes in the namespace, setting and getting node data, setting watches on nodes to receive notifications of changes, and performing various other operations such as listing child nodes, checking node existence, and setting access control lists.

The code is organized into several modules, including a main client module (zookeeper.c), threading adaptors (st_adaptor.c and mt_adaptor.c), hashtable (zk_hashtable.c) and various utility modules for data serialization, buffer management, logging, and error handling.

Functions in zookeeper.c handles communication with the ensemble using ZooKeeper’s wire protocol and manages the client’s session state, including connection management, session timeouts, and error handling. It also supports SSL/TLS connections to ZooKeeper servers through the use of OpenSSL.

A short rundown of the main functions defined in zookeeper.c:

• zookeeper_init(): called to initialize the ZooKeeper client library. It takes a set of connection parameters, such as the host and port of the ZooKeeper ensemble, a timeout value, and a callback function to handle connection state changes. The function returns a zhandle_t handle, which is used to identify the client’s session.

• zookeeper_close(): called to close the client’s session with the ZooKeeper ensemble. It takes a zhandle_t handle as input and returns 0 on success.

• zookeeper_create(): used to create a new node in the ZooKeeper namespace. It takes a zhandle_t handle, the path of the node to create, the initial data for the node, a set of flags to specify node creation options, and a callback function to handle completion of the operation. The function returns the path of the newly created node on success.

• zookeeper_delete(): used to delete a node from the ZooKeeper namespace. It takes a zhandle_t handle and the path of the node to delete, along with a version number that must match the current version of the node. The function returns 0 on success.

• zookeeper_set(): used to set the data associated with a node in the ZooKeeper namespace. It takes a zhandle_t handle, the path of the node to set, the new data for the node, and a version number that must match the current version of the node. The function returns the version number of the newly set data on success.

• zookeeper_get(): used to get the data associated with a node in the ZooKeeper namespace. It takes a zhandle_t handle, the path of the node to get, a callback function to handle completion of the operation, and a set of flags to specify node retrieval options.

• zookeeper_wget(): similar to zookeeper_get(), but it also registers a watch on the node to receive notifications of changes to the node’s data.

• zookeeper_exists(): used to check if a node exists in the ZooKeeper namespace. It takes a zhandle_t handle, the path of the node to check, a callback function to handle completion of the operation, and a set of flags to specify node existence check options.

• zookeeper_wexists(): similar to zookeeper_exists(), but it also registers a watch on the node to receive notifications of changes to the node’s existence status.

mt_adaptor.c provides a multithreaded adaptor layer to allow client applications to use the ZooKeeper library in a multithreaded environment. The module uses a global lock to protect access to the ZooKeeper library’s internal state, and all calls to the library are made through the adaptor functions provided by mt_adaptor.c.

zk_hashtable.c provides a hashtable implementation for storing key-value pairs and uses chaining to handle collisions. Each entry in the hashtable is a struct hash_node, which contains a key-value pair and a pointer to the next node in the chain. Data typically stored in this table are zhandle_t, znode_t watcher_registration_t and watcher_registration_t.

I forked the repo to explore the code and added detailed comments about the code flow, session management, error handling etc. (just for learning reference, not meant to be production grade) so I can always refer to in case of doubts. Full diffs of the change is available here.

Writing Code: ChatGPT has been incredibly helpful in writing small Bash and Python scripts and tools. All I have to do is provide the requirements and it spits out beautiful code solving the given problem. The clarity of the answer depends on the clarity of the question, which is why I appreciate how it forces me to think and explain the requirements in simple terms. I also use it to write starter code, class definitions, templates and unit tests.

Summarizing Code: I often use ChatGPT to summarize long lines of code. This makes it easier for me to understand and troubleshoot, especially when dealing with complex codebases. I do this only on opensource code though.

Writing queries, commands: Generating jq queries for complex data can be time-consuming, but ChatGPT can generate them quickly and accurately. I also find it useful to write shell commands that otherwise required me to read the man pages and find the options and syntax by myself.

Generating Test Data: ChatGPT can create test data in formats like JSON and YAML, making it a great tool for manual data entry.

Proofreading Content: ChatGPT can help me check content for language, grammar, and correctness. This is incredibly helpful, especially when I need to write professional emails, feedback, or reviews. (p.s. this post also went through this exercise 😄)

Creative Writing: I also use ChatGPT to make up stories with different characters and themes to tell to my kid.

I briefly tried the new Bing with ChatGPT and Neeva, but I found that the user experience and result correctness weren’t as good as ChatGPT.

I don’t know much about the AI technology behind ChatGPT, but I am super optimistic about the future powered by tools like ChatGPT. I believe that ChatGPT has the potential to revolutionize the way we work and communicate. It simplifies tasks, improves efficiency, and challenges us to think more clearly and communicate more effectively. ChatGPT has become an essential tool in my daily life, and I’m so excited to see how it will continue to evolve in the future.

analytics:
  provider: "google-universal"
  google:
    tracking_id: "UA-908945612-1"

To move to GA-4, I had to

• update the provider to google-gtag
• get the measurement ID from the data stream (as explained here)

analytics:
  provider: "google-gtag"
  google:
    tracking_id: "G-TYVCJUF74I"

Post update, data from the universal analytics tag didn’t seem to be carried over. GA-4 property shows only the data from the new traffic. The home page for GA-4 property does look better than the old one though, with cleaner UI, personalized dashboard and customized insights and recommendations.