I had written this:

llm = ChatOpenAI(
    model="gpt-4.1-mini",
    temperature=0,
    top_p=1.0,
    max_tokens=512,
    timeout=30,
    max_retries=2
)

…and I only understood few of those parameters (and didn’t even know the others existed).

The rest values were generated by AI.

So I decided to understand every single decision I (my AI) had made. This post is that explanation—the version I wish I had read before building anything with LLMs.

The Configuration

llm = ChatOpenAI(
    model="gpt-4.1-mini",
    temperature=0,
    top_p=1.0,
    max_tokens=512,
    timeout=30,
    max_retries=2
)

Every one of these changes how your application behaves in production.

Temperature: Creativity vs Consistency

This was the first parameter that actually clicked for me.

temperature controls how random the model’s output can be.

Imagine asking:

"What is the capital of France?"

With temperature = 0, the model will almost always answer:

Paris.

With temperature = 1.0, the model may still answer Paris, but the wording can vary:

• The capital of France is Paris.

• Paris is the capital city of France.

• France’s capital is Paris.

Now imagine a different prompt:

"Write a startup idea involving drones."

Temperature becomes much more important.

Temperature = 0

A drone-based package delivery optimization platform.

Temperature = 1.2

A swarm of autonomous drones that build temporary communication networks during natural disasters.

Same prompt, different level of exploration.

For my project, I wanted deterministic financial explanations, not creative storytelling.

So I chose:

temperature = 0

Because if a user asks the same financial question twice, I want nearly the same answer twice.

Top-p: How Many Candidate Words the Model Considers

This parameter confused me for a while because it sounds similar to temperature.

It is different.

The model predicts probabilities for the next word.

Example:

With:

top_p = 1.0

the model can consider the full probability distribution.

With:

top_p = 0.8

it only considers the most probable words whose cumulative probability reaches 0.8.

That means low-probability words are discarded before sampling happens.

A useful intuition:

• Temperature changes randomness

• Top-p changes the pool of possible choices

In practice, most applications tune either temperature or top_p, not both aggressively.

I kept:

top_p = 1.0

because I was already controlling randomness with temperature.

Max Tokens: Controlling Response Length

This parameter is surprisingly practical.

max_tokens = 512

means the model can generate at most 512 output tokens.

Think of tokens as chunks of text.

Roughly:

• 1 token ≈ 0.75 English words (very approximately)

So 512 tokens is usually a few hundred words.

Why does this matter?

Without a limit, a model may produce unnecessarily long responses.

For example:

Prompt:

"Explain compound interest."

Unlimited response:

• definition

• formula

• history

• examples

• investment strategies

• taxation discussion

• long conclusion

Limited response:

• definition

• formula

• short example

For an application, shorter answers often mean:

• lower latency,

• lower cost,

• more predictable behavior.

I did not need essays.

I needed concise financial explanations.

Timeout: Don’t Wait Forever

This is one of those parameters that matters only when things go wrong.

timeout = 30

means:

If the model does not respond within 30 seconds, stop waiting.

Imagine:

• network issues,

• API congestion,

• provider delays,

• temporary outages.

Without a timeout, a request might hang indefinitely.

In a web application, that means users keep staring at a loading spinner.

A timeout lets your application fail gracefully and return an error or retry.

It is not about the model.

It is about protecting the user experience.

Max Retries: Surviving Temporary Failures

APIs fail.

Connections drop.

Rate limits happen.

Transient server errors happen.

This parameter tells the client:

max_retries = 2

If the request fails due to a temporary issue, try again up to two additional times.

Example:

Attempt 1 → timeout

Attempt 2 → network reset

Attempt 3 → success

Without retries, that request would have failed even though the service became available moments later.

Retries are one of those production engineering habits that appear everywhere:

• databases,

• message queues,

• HTTP services,

• cloud APIs,

• LLM APIs.

They improve reliability without changing your business logic.

Why These Choices Fit My Project

My project was an AI financial assistant.

The goal was not entertainment.

The goal was accurate, repeatable explanations.

So my configuration looked like this:

What looked like random numbers started looking like engineering decisions.

Where I Would Use Different Values

Creative writing

temperature = 1.2
top_p = 0.95
max_tokens = 1500

Useful for stories, poems, marketing copy, and brainstorming.

Code generation

temperature = 0
top_p = 1.0
max_tokens = 800

Consistency matters more than novelty.

Customer support chatbot

temperature = 0.2
top_p = 1.0
timeout = 10
max_retries = 3

Fast, reliable, and mostly deterministic.

Research assistant

temperature = 0.4
top_p = 0.9
max_tokens = 2000

Allows slightly broader reasoning and longer explanations.

The Lesson That Actually Stayed With Me

Before this project, I used LLM parameters the same way beginners use Redis commands or Docker flags.

Copy.

Paste.

Hope.

After spending a few hours understanding them, I stopped seeing them as magic numbers.

Now when I read:

temperature=0

I immediately think:

“This system wants deterministic behavior.”

When I read:

max_tokens=128

I think:

“Someone optimized for latency and cost.”

When I read:

max_retries=5

I think:

“Reliability mattered in this application.”

That was the real shift.

I didn’t just learn what these parameters do.

I learned to read an LLM configuration and understand the design priorities behind it.

And that feels a lot more useful than memorizing another API call.

while (true) {
  await fetch('/api/order/create');
}

Your server has no idea this is a bug. It just sees traffic. CPU usage spikes, your database starts struggling, other users experience latency, and your cloud bill gets ugly. Replace the bug with a malicious user trying 10,000 passwords against /login, and the story gets worse.

That is the entire reason rate limiting exists.

A rate limiter answers one question before your business logic runs:

Is this user allowed to make another request right now?

If yes, continue.

If no, return HTTP 429 (Too Many Requests).

I used to think this was one of those backend topics every framework had already solved. Then I tried designing one from scratch, and it broke in more places than I expected.

This post is that journey — mistakes included.

Attempt 1: Just Use a HashMap

The rule I picked was simple:

100 requests per minute per user

My first instinct was a hash map.

userId → count

Constant-time lookups. Trivial to increment. On a single server, this genuinely works.

I was fairly pleased with myself for about five minutes.

Where It Broke: There Is No “The Server”

Real systems rarely have a single server.

A typical deployment looks like this:

Client
   |
Load Balancer
   |
+---------+   +---------+   +---------+
| Server A|   | Server B|   | Server C|
+---------+   +---------+   +---------+

If user123 sends 100 requests, the load balancer may distribute them across all three servers.

• Server A sees 30

• Server B sees 40

• Server C sees 30

None of them knows the real total.

This was my first distributed-systems lesson:

The counter cannot live inside the application process.

It has to live somewhere every server can access.

Why Redis Instead of PostgreSQL?

My next question was obvious.

Why not store the counter in PostgreSQL?

Technically, you can.

But rate limiting is a write-heavy, latency-sensitive workload. Every request needs to update a counter before any business logic executes.

Redis is a much better fit because:

• it stores data in memory,

• counter operations are extremely fast,

• keys can expire automatically using TTL,

• and it is commonly used for ephemeral state such as caches, locks, and rate limiters.

So my design became:

rl:user:123 → 57

stored in Redis.

Where It Broke Again: The Race Condition

At this point I thought I was done.

Then I imagined two servers receiving requests from the same user at the exact same moment.

Server A:

Read count = 99

Server B:

Read count = 99

Both increment.

Both allow the request.

The user just made 101 requests even though the limit is 100.

The problem was not Redis itself.

The problem was my sequence of operations.

1. Check if the key exists

2. Create it if necessary

3. Read the count

4. Compare with the limit

5. Increment

6. Return allow/reject

Across multiple servers, those operations are not atomic.

The Fix: Push the Whole Operation Into Redis

This is where Lua scripting finally made sense to me.

Redis executes commands sequentially, and a Lua script is executed atomically. The entire script completes before another command can modify the same data.

Instead of sending multiple Redis commands from Node.js, the script can perform everything in one step:

• increment the counter,

• create the key if necessary,

• set the expiration time,

• compare against the limit,

• return allow or reject.

No other request can observe the counter halfway through the operation.

The race condition disappears.

I Started With Redis Hashes

My initial design used a Redis Hash.

user123 → count

Later I realized something interesting.

For a single numeric counter, a Redis String is actually simpler.

rl:user:123 = 57

Redis can increment strings directly, so there is no need for a hash unless I want multiple fields such as count, plan, lastSeen, etc.

It was a small lesson, but an important one:

The simplest data structure that satisfies the requirement is usually the best one.

Where It Broke a Third Time: The Fixed Window Problem

At this point I had a rate limiter that worked correctly across multiple servers.

Then I noticed this.

A user sends:

• 100 requests at 12:00:59

• another 100 requests at 12:01:00

They successfully make 200 requests within two seconds.

The counter resets exactly on the minute boundary, so the algorithm has no memory of what happened one second earlier.

This algorithm is called a Fixed Window Counter.

It is fast and simple, but it has boundary problems.

Attempt 2: Stop Counting, Start Remembering When

Instead of storing only a counter, I started thinking about storing the timestamp of every request.

Suppose requests arrive at:

• t = 0

• t = 1

• t = 3

If the window size is 5 seconds, then at t = 5 the request from t = 0 has expired.

Only the requests from t = 1 and t = 3 still count.

The rate limiter is now evaluating the last N seconds, not the last calendar minute.

A common implementation stores timestamps in a Redis Sorted Set (ZSET).

For every request:

1. Remove timestamps older than now - window

2. Count the remaining timestamps

3. If below the limit, insert the current timestamp

4. Otherwise reject the request

This is the Sliding Window approach.

It almost completely removes the fixed-window boundary problem.

The Tradeoff I Didn’t Expect to Care About

I initially assumed sliding window was simply “better.”

It isn’t.

It is more accurate, but it is also more expensive in memory and CPU.

For many production APIs, fixed windows are still a perfectly reasonable choice because operational simplicity often matters more than eliminating a rare edge case.

Where This Design Still Fails in Production

I thought the design was finished.

Then I imagined one enterprise customer sending 50,000 requests per second.

Every request updates the same Redis key:

rl:user:enterprise123

Now every Node.js server is hammering the same Redis key.

Redis executes commands sequentially, so all requests for that key are effectively serialized through the same Redis shard.

The architecture becomes:

Many Node.js servers
          |
          v
    One Redis key
          |
          v
 Serialized execution

At moderate scale, this is fine.

At very high scale, that key becomes a hot key bottleneck.

The bottleneck is no longer my application.

It is Redis itself.

How I Would Improve It Next

I have not implemented these yet, but these are the directions I would explore next:

• Redis Cluster to distribute data across multiple shards

• Sharded counters where one logical counter is split across multiple keys

• Token Bucket algorithms for burst-friendly traffic

• Leaky Bucket algorithms for traffic smoothing

• Local in-memory caching for extremely hot keys

• Multi-region rate limiting for globally distributed deployments

I intentionally stopped before these optimizations because I wanted to understand one production-ready version deeply before making it internet-scale.

What Actually Stuck With Me

I started this exercise thinking a rate limiter was just “store a number somewhere.”

It turned into a lesson about:

• shared state across servers,

• atomicity,

• expiration semantics,

• algorithm tradeoffs,

• and bottlenecks that only appear once you stop imagining toy traffic.

That buggy while (true) loop from the beginning of this post is still the entire point.

Every layer I added — Redis, Lua, sliding windows — exists because somewhere, someone’s client is going to do exactly that, and your server needs an answer ready before it happens.

I don’t think this is the final version of a rate limiter.

I think it is the first version I’d be comfortable defending in a backend interview.

And that, for me, was the interesting part.