Learn With Sundar

Software architects today have access to an incredible number of technologies.

Microservices, Kubernetes, Serverless, Kafka, NoSQL, Event Driven Architecture, Service Mesh, Cloud Native services, AI, LLMs, Vector Databases, and many more.

Every year, the list gets longer.

For architects and engineers, this is exciting. We have more tools, more patterns, and more ways to solve problems than ever before.

At the same time, it creates a different kind of challenge.

With so many technologies available, how do we decide what we actually need?

Learning a Technology and Needing It Are Different Things

As technology professionals, continuous learning is part of our job.

We should understand new technologies, experiment with them, and learn the problems they are designed to solve.

The more we learn, the larger our architectural toolbox becomes.

But there is an important difference between:

“I understand this technology.”

and

“My system needs this technology.”

Knowing Kubernetes does not mean every application needs Kubernetes.

Understanding Microservices does not mean every system should be divided into Microservices.

Knowing Kafka does not mean every asynchronous workflow needs an event streaming platform.

And understanding AI and LLMs does not mean every application needs an AI component.

This is where architectural judgment becomes important.

Start With the Problem, Not the Technology

When starting a new system, it is easy to jump directly into technology discussions.

Should we use Microservices?

Should we deploy on Kubernetes?

Should communication be event driven?

Should we use Kafka?

Which database should we choose?

Where can we introduce AI?

These are useful questions, but they should not be the first questions.

The first question should be much simpler:

What problem are we trying to solve for the customer?

Once we understand the problem, we can start asking architectural questions.

What scale do we expect?

What availability does the business require?

What are the performance expectations?

What are the security requirements?

How quickly does the system need to recover from failures?

How frequently will the system change?

Can the team operate this architecture effectively?

What will it cost to build, run, monitor, and maintain?

These requirements should influence our technology decisions, not the other way around.

Microservices Are Powerful, but They Come With a Cost

Microservices are a good example.

There are many situations where Microservices make sense.

We may need independent deployments.

Different parts of the system may need to scale differently.

The business domain may have clear boundaries.

Multiple teams may need to work independently.

In those situations, Microservices can provide significant value.

But Microservices also turn many in process interactions into network interactions.

Now we need to think about service discovery, network failures, retries, timeouts, distributed tracing, API contracts, data consistency, deployment coordination, observability, and many other concerns.

The architecture may still be the right choice.

But the complexity should be justified by the problem.

If the application does not require those characteristics, a well designed modular application may solve the same business problem with much less operational complexity.

Choosing the simpler approach does not mean we do not understand Microservices.

Sometimes it means we understand them well enough to know that we do not need them.

The Same Question Applies to Kafka

Kafka is another powerful technology.

If a system needs high volume event processing, asynchronous communication, loose coupling, event replay, or integration across multiple systems, Kafka can be an excellent architectural choice.

But imagine that the application has only a few simple asynchronous workflows.

Do we really need to introduce an event streaming platform?

Maybe we do.

Maybe we don’t.

The important thing is that the decision should come from the requirements.

The same thinking applies to Kubernetes, Serverless, NoSQL, Service Mesh, GraphQL, AI, LLMs, Vector Databases, and almost every other technology we evaluate.

Every Technology Solves Problems and Introduces Responsibilities

One question I find particularly useful when evaluating technology is:

“What problem will this technology solve for us, and what complexity will it add?”

We often spend more time discussing the first part than the second.

But both matter.

A new technology can mean more infrastructure to operate.

More things to monitor.

More failure scenarios to understand.

More security considerations.

More skills for the team to develop.

More dependencies to manage.

More production issues to troubleshoot.

And sometimes, more cost.

None of these automatically make the technology a bad choice.

They simply become part of the trade off.

If the value it provides justifies the additional complexity, it may be exactly the right decision.

If it does not, perhaps the simpler solution is better.

Complexity Has a Cost

Architecture diagrams can make complex systems look surprisingly clean.

A few boxes.

Some arrows.

A message broker in the middle.

A cluster around the services.

Maybe an API Gateway at the top.

On the diagram, everything looks organized.

Production is different.

Every box on an architecture diagram represents something that eventually needs to be deployed, secured, monitored, upgraded, scaled, debugged, and supported.

Every arrow represents communication that can become slow, unavailable, duplicated, delayed, or fail completely.

This does not mean distributed architecture is bad.

It means distributed architecture should exist for a reason.

Complexity is sometimes necessary.

Accidental complexity is not.

Don’t Build Tomorrow’s Architecture Too Early

Another common challenge is designing too far into the future.

We want our systems to scale.

We want them to be flexible.

We don’t want to rewrite everything when the business grows.

These are valid concerns.

But there is a difference between designing for change and designing for every possible future scenario.

We may build a system today for millions of users when the product currently has thousands.

We may introduce Microservices because the organization might eventually have dozens of development teams.

We may build a sophisticated event architecture because someday the application might process huge volumes of events.

That future may happen.

Or it may not.

Meanwhile, the team has to live with today’s complexity every day.

A better approach is often to create an architecture that meets today’s requirements while keeping reasonable options open for tomorrow.

Create clear boundaries.

Keep modules well separated.

Define contracts carefully.

Document important architectural decisions.

Measure what actually happens in production.

Then evolve the architecture when the requirements provide a reason to evolve it.

Architecture Should Evolve With the Problem

Architecture does not have to predict the future perfectly.

It needs to support change.

A system that starts as a modular application can later separate certain domains into independent services when there is a real reason.

A simple asynchronous mechanism can evolve into an event streaming platform when throughput and integration requirements increase.

A straightforward deployment model can move toward more sophisticated orchestration when operational scale demands it.

This is why architecture should be treated as something that evolves with the business rather than something we design once and consider finished.

Production gives us information that architecture meetings cannot.

We learn where the real bottlenecks are.

We learn which components change frequently.

We learn where scaling is actually required.

We learn which dependencies cause failures.

And we learn what customers really need.

Those signals should influence the next architectural decision.

Keep Learning, but Choose With Purpose

None of this is an argument against modern technology.

In fact, I believe the opposite.

Architects should continuously learn.

Learn Microservices.

Learn Kubernetes.

Learn Kafka.

Understand distributed systems.

Explore Serverless.

Understand different database models.

Learn what AI and LLMs can bring to software systems.

Experiment with new tools and patterns.

The broader our knowledge, the more options we have when solving a problem.

But our value as architects is not measured by how many technologies we can place on an architecture diagram.

It is measured by whether we can choose the right ones for the problem in front of us.

Sometimes that means introducing a sophisticated new technology.

Sometimes it means continuing with something simple that already works.

And sometimes one of the best decisions we can make is:

“We understand this technology, but we don’t need it right now.”

That is not resistance to innovation.

It is a conscious engineering decision.

Final Thoughts

The technology landscape will continue to grow.

There will always be another framework, platform, architecture pattern, database, cloud service, or AI capability to learn.

That is one of the things that makes software engineering interesting.

We should stay curious.

We should keep learning.

We should experiment.

But when it comes to production architecture, we should always come back to the fundamentals.

What problem are we solving?

What does the customer actually need?

What are the important quality attributes?

What trade offs are we making?

What complexity are we introducing?

Can we solve the problem in a simpler way?

Modern architects have more tools than ever before.

Our challenge is no longer finding a technology that can solve the problem.

Our challenge is knowing which technology is appropriate for the problem, and which ones we can leave out.

Because in the end, good architecture is not about making the architecture bigger.

It is about making the solution better.