It was a once in a lifetime opportunity, an opportunity to learn how to write a Search engine from the ground-up. Working with some of the best and brightest engineers, incubating innovative technologies, to architect, to develop and to finally ship the product to millions of users.
It took us about 2 years growing Maguro from the pre-baby to baby stage and the technology is still pretty much alive powering Bing search.
It was a tremendous undertaking, it was a journey that requires in-depth technical skills. However technical skills alone does not complete the project. The success of the project also depends on many non-technical things. I would like to share 5 invaluable lessons from incubating Maguro.
1. Have Ambitious but Achievable Goals
We were starting the project with the ambition synonymous to sending one to travel to different galaxy (a.k.a moonshot project). It does not sound even real. It is exciting to think that way. However when you have constraint and you have to deliver something, you need to balance the ambitious goals with setting achievable goals.
Often the best way to think about setting the goal is to have a view of the exit, then backtrack from there.
2. Healthy Team Dynamics
Managing a large team is no easy task and putting too much processes can hamper progress. Team members that cannot work together do more harm than good. Having a large team with all stars players does not guarantee a success. A team of three can be magical. We were a team consist of three engineers plus a researcher. Each of us specializes on three different important aspects in any software development like:
- Full stack engineer; ability to work at each different level of the tech stack and pulling everything together and getting things done and make sure the system is always running and healthy.
- Hardcore engineer; that can squeeze the performance to the metal and so laser focused on getting the maximum performance out of the system; making sure that the team adopting the right architecture that can scale up and out.
- Team lead/Generalist engineer; a good engineer that also has very good communication skills. Someone that can interface with the rest of the world, communicating our progress and pulling out ideas from outside; shielding the team from a lot of unnecessary overheads and randomization; making sure that the team is making all good trade-offs, is on track for the milestones and executing on the right direction.
- Someone from research; that has the vision and at the same time knows what to build.
“Context switching is overheads and should be avoided if possible.”
You want to distribute tasks in a way that it reduces overheads and can be executed in the most efficient way. Work is already challenging, we do not want other things to hamper the progress. We want everything else to help the team to focus on what matters.
Life can be very stressful with a lot of expectations and deadlines. There are always good and bad times. Team happiness is one very important aspect of running a project. It pulls the team energy together and it enables the team to overcome challenges together.
3. Fail-fast, Iterate and Retry Attitude
We often do not have answers to everything all at once. It is quite dangerous to spend a lot of energy and resource on one thing that is not proven, and you should always give a little room for possibilities that such an option can fail.
It is probably wise to have a mindset that plan A might fail and to be ready with plan B and plan C.
4. Use the right technologies
Picking the right technologies is not less important. We started with C# until we hit a lot of perf problems. Thinking that it is an easier language to write with better productivity tools supporting it, such as better unit-testing support, better intellisense, faster build with Visual Studio. The reality is that some of the C# code can be optimized close to C++ perf some cannot. Ironically, some of the optimized code looks like a C/C++ code, and it defeats the purpose of using C# in the first place.
Some low level IO API was not available in C#, so we were never able to hit perf close to do hardware performance that was only possible with C++ implementation. Using C++ we were able to build index by running a screaming performance of merge close to the disk performance specification ~ 100-125Mbytes per second.
5. Goal-Oriented Execution Plan
There are so many things to be done. Using conventional process managing incubation project might not work, we tried and always failed. Different from running daily production system where Scrum or Kanban might work better. Especially early incubation process requires a different dynamics and flexibilities.
We like to do a lot of mini experiments. Fail-fast, modify and retry. Sometimes we require some silent from any morning regular meeting; we often work till late or even doing a lot of work from home. What work for us was a light-weight process where we create milestone by defining the delivery goals.
- next month is about getting single box running end to end;
- the following month is about running 4 boxes end to end.
It is simple that everyone can understand the goal. Coupled with driven developers and get things done attitude and everybody is happy and has the same view of high level picture about what to deliver, you can focus on what matters and less on overheads.