DevOps

DevOps - rulez!

TBD

Working in Trading infrastructure, we are designing CI Pipeline for the comprehensive testing of the high-performance load balancer written on GOlang. We mainly focus on the performance and the stability of the product.

To meet those reqiurements we created next stages:

• Race condition detector
• Core dump detector
• Performance degradation detector
• Resource degradation detector(cpu, heap, runtime in our case)

That helped us significantly improve the time of delivery and the quality of the product.

Inteligent Load Balancer

Bid infrastructure

I'm in the team which is doing an Intelligent Load Balancer.

That smart server is able to keep more than 1.5K requests per one core per second.

Here we focus on the performance tasks and doing hard work to bring new features and keep performance at least at the same level as it is on the previous version.

We are also building CI/CD pipeline which allows us to keep development velocity, high quality, and delivery speed.

It's build over Jenkins CI and includes some interesting and non-standard stages like "profiles comparison" among the standard tasks.

Here we're following the "Configuration as a code" approach and also do not couple the main functionality with the Jenkins Groovy.

That allow us to use same configuration across all the host machines. Which is very convenient.

Another focus of our team is so called "Performance-Playground" component.

That is set of the configurations which are orchestrated by Terraform.

That allows us to set up a dev mini-cluster with the prod environment

and replay the any slice of the live traffic through the development branch; collect profiles, metrics and compare them with the prod values.

Besides that, we are working with the other teams to insert the load balancer in their existing testing frameworks.

The main challenge here is that current framework is a mature project with the huge codebase. Much complex logic has already built over that framework and many more tests are written. The task is to migrate all without affecting the current processes.

It sounds like "It's just a proxy server, you only need to change the system ports". Sounds good, but there are some nuances :)

Microservices

Of cause we tried them

When maintaince of monolith becomes too hard it's time to go to microservices architecture.

We started breaking our monolith into microservices and as a result we designed API Gateway with authZ inside it and several services behind the gate.

All that infrastrucute was protected by the CI pipeline, which was able to startup the whole infrastructure and run the tests agains it as well as to focus only on one particular service and play with it.

I started to like mSOA, when some time later I was able completely rewrite one service. That work was done fast and without any dependency to the rest of the system.

Rest API

We created a monolith application server which was designed the same way as a "LEGO" constructor. That means that it has a lot(I mean a lot)of small components.

Our primary focus was on the possibility of the system to configure itself on the fly by given configuration. That configuration driven development allows us to manage objects and thier relations with OpenAPI compatible schemas.

The schema was the hart of the system.

Such an approach helps us

• speedup the projects delivery
• keep all souce code in one repository
• support many projects by small team

That was a "rocket scince" system. Our team becomes an experts in Python, Django and DRF.

We also build a CI pipeline which allowed us to keep high quality of the delivered projects.

Social network

That was fun!

I've created a social network for the pets and their owners.

That was time of social networks, it appeared one by one, I think, every month. Our MVP had a standard set of features:

• News feed
• Gallary
• Account management
• Messaging
• Community
• Posts management

That time I did the full-stack and actually was impresed how cool things you can do with JavaScript(that time I use jQuery and Bootstap).

Email marketing Service

That was all about emailing. We sent around 30-40 millions of emails every day and every email was personalized. I think the most challanged tasks were: the Mail sender and the Targeting machine.

The final version of Mail sender was written on Perl programming language and was based on event loop programming approach.

That was time of "hardcore": daemons, multiprocessing communications, sockets, synchronizations and all that stuff.

Targeting machine was my first experience with AdTech. We did personalized targeting for each letter.

Ultimately, we came up with the solution which allowed us to check all possible variants and get the result with O(1). Basicaly, we created the complext data structure with custom hashing function. When ads were ready, we generetead AdCache, that was really huge, but worked super fast.

By the way, doing that project I learned TDD and since that time I do testing for all my code everytime. Actually, that was time when many current technologies were born:

• jQuery was wolking around the world and ajax was like a magic
• Hudson, these days we call it Jenkins, made automation an easy task
• The Django framework made making web an easy task
• Python was beating Perl
• Node showed that JavaScript is cool for backend
• Golang showed how it's easy to make background processes and orginize communication channel between them.
• and many more...

Actually, I am happy that I participated that technology revolution, and still participating.