System Design - Over Engineering
Building software or a business is an iterative process. One that requires you to run tests, adjust for errors, and test again. However, losing track of the end-goals in this process can be costly and prove to be more detrimental to future successes. One such miscalculation is overengineering.
In a bid to build a system that is highly scalable and fault-tolerant, you may fall into the trap of creating more complex layers in the system than is necessary. You may end up solving problems that are not relevant to the business requirements. Potentially, creating new problems that will suffice in the future. Overengineering can increase development and maintenance expenses, delay product release, waste time and money, and hinder positive user experiences.
When developing a software system or product, you should look to start and maintain a simple, fast, and functional iteration of your concept—something that works and meets the clients’ and end-users’ demands. Once you have done that, you can scale up from there but it’s important to never lose sight of the core concept. This recommendation helps you avoid several issues that startups and developers tend to encounter.
What is Overengineering?
Overengineering is the practice of developing a solution to a problem in a complicated manner where a more efficient approach can be used. Overengineering usually always introduces new layers of abstraction and complexity into your product than is necessary. It involves coming up with a complicated solution or product design, especially one that goes beyond your immediate demands.
Different systems require different designs, some of them more complex than others.
For example, if you’re building a mobile banking application, you’ll have different levels of abstraction for multiple complex features, which can include risk management, service diversification, and shared member interaction.
A bank application most likely requires the following:
Bank Account Management
Advanced Security and Fraud Alerts
QR Code Payments
New account enrollment and existing account management
Deposits and withdrawals
Loan management
Interest rate calculation
Introductions to new financial products
Record maintenance
Mobile Check Deposits (MCD)
Peer-to-Peer Payments
Bill Payments
Intelligent Chatbots
Gamification and rewards
In comparison, the route to success in building a single player game app that accepts one form of payment is more direct. Applying similar payment features as our banking example to a gaming app would be quite fancy but unnecessary, and clearly, an overengineering solution. Solving the core requirements saves time and money. Investing to develop simple steps for user registration, player management, user dashboards, and save options would be more sensible solving.
In our extreme example, it’s easy to see that adding payment features required for a banking application to a game application, only because we need payment processing would be an overkill. Another overengineering disaster. Every feature and layer of development should be unequivocally justified.
Is Overengineering a Big Problem? Why Does It Happen?
When designing any software system with a customer’s needs in mind, it’s easy to brainstorm almost every possible problem. Your session turns into hours of tasks for software developers. We understand, there’s a duty as a service provider to care about your customers. However, going beyond the immediate needs of the customer causes you to go off-track and potentially lose that very customer.A good reminder is contemplating immediate problems from the phase or feature you’re currently developing. That is, if point A is the Save screen on a game app, document which solutions are more wanted by users. Fulfill those user needs first and move to the next one.The intent behind overengineering is often noble; the world has become a global village, and you need not think about just your immediate market. You should consider your competition, deadlines, and the uncertainty of building a system while continuously delivering value.Now more than ever, companies have to “build fast and fail fast,” assuming the failures are small but corrected. This disillusioned way of thinking may cause companies to overdo it with features in the hopes of building and satisfying strong market demand. The pressure compounds for startups and small firms to iterate faster and compete, given that the development for the main product of one company, may already be an existing software in the family of products for a larger company.
Examples
To understand this better, consider Zoom, an innovative communication platform presently valued at around $3B. Zoom had unprecedented success during the pandemic and was almost instantly the number one cross-communication platform in the world but as grand as that is, that product is just one of many that’s included in the Google workplace.Google workplace is a suite of softwares comprising Gmail, Google Drive, Google Docs, Google calendar, Google Meet, and many others. Now, there is an added advantage for companies like Google, if their attempt at building a product fails, they can acquire a smaller company who has succeeded in building the same product or maybe even kill the smaller company completely.A better example exists when we talk about novelty features in software applications. Take for instance, Snapchat vs Facebook.Sometime in 2015, Facebook, the company that owns Instagram, was concerned that Snapchat was poaching some of its users. Snapchat was becoming more popular among social media users as a result of its iconic Stories feature. Users could instantly add filters and effects to photos and make them more realistic or candid. Sharing was likewise quick and easy. The ability for the stories to disappear after some time was the best feature.In August 2016, Facebook unveiled Instagram Stories, a direct replica of Snapchat Stories that even used the same name. Instagram Stories overtook Snapchat’s daily active user count in just eight months. In the first year of Instagram Stories, an extra 250 million users were added, while the value of Snapchat’s stock plummeted significantly.
How to Avoid Overengineering? – Approach System Design Using the Simplest Form Possible.
When adding new layers of abstraction to your system, you need to ask yourself if this move will introduce more complexity into the system. If yes, does it meet your business requirements? Then, you need to think carefully about the benefits of this increased complexity and if it is worth the cost.Overengineering has its cost. With every new concept added to software, there are new tests and actions to draw out. Things like distributed logging, monitoring, inter-service communication, action delivery markers, alerts, tracing, data pipelines, and technical health checks. When designing a complex system, you need to understand the trade-offs involved and examine potential bottlenecks because there is no perfect solution.You need to understand your use case, the customer’s needs, and the architecture required to meet those business requirements. Otherwise, you will have a system that is both slow and more complex, and it will also cost more to build, test, and maintain.
Focus on MVPs
A minimal viable product (MVP) is a development strategy where a new product or website is created with only core features to appease early users. The MVP allows the product team to gain insights into the customers’ needs, allowing them to iterate and improve the product as quickly as possible with the least effort.By focusing on providing an MVP, developers may avoid tedious and ultimately labor. Instead, they test and validate presumptions about a product requirement by iterating on functioning versions and responding to feedback. This approach is beneficial since it provides more precise feedback than a prototype.The benefits of an MVP approach include the following:
Products can be ready for release quickly
It informs developers as to whether the core features are functioning as intended
It saves time, effort, and money during the development stage
It can provide a much clearer answer as to whether the product is viable
It lowers risk by eliminating expensive errors
It can advance the project for a relatively low outlay
Refactoring: Improving the Design of an Overengineered System
Code refactoring is the systematic process of restructuring and improving existing computer code without changing its functionality. Refactoring aims to enhance the software’s non-functional attributes—its design, structure, and implementation—while keeping its functionality intact. This process enhances code readability and reduces complexity; this creates a cleaner, simpler, more expressive internal architecture. Refactoring makes it easier to maintain the source code and improves the system’s overall performance by making it faster and less memory intensive.
Software with a simple architecture is most appropriate for use cases where the app’s requirements are straightforward and the application is not overly complex. Examples include a to-do list app, a sports news app, a company’s internal tax calculator app, or a comparable open-source program. These are the use cases where the company is confident that the program will have few features, be straightforward, and not need any significant expansion.
You should understand that refactoring and re-writing code come at a hefty expense. It takes a lot of time and effort to deconstruct features from different complicated architectures and re-implement them in a straightforward, tightly coupled architecture. Choosing a simple or complex architecture largely depends on our use case. Before choosing an architecture, I advise keeping things simple and thoroughly understanding the requirements in great detail.
Let’s consider a real-life example. Istio is an open-source dedicated infrastructure layer that enables you to transparently connect, observe, manage, and control your applications without interfering with the code. Istio recently transitioned from microservices to a simplified, independent architecture. According to them, using a monolith architecture made creating value easier and increased usability and operability characteristics. The complexity of the microservice architecture was disproportionate to the value it offered and didn’t meet their business requirements.
Conclusion
In this article, we introduced the concept of overengineering, explained why it happens, why it should concern you, how to focus on getting a minimum viable product, and how to solve by refactoring your software. Take the time to understand your business requirements that will help you design a system focusing on getting to the minimum viable product. Endertech can assist you in creating a scalable, reliable, and fault-tolerant system if you need consulting.