[Development Environment Improvement Retrospective] The importance of DX(Developer Experience) that boosts productivity

‍

Hi, I'm Hamill, Ntuple backend developer who thrives on improving the experience!😀 Today I'm going to talk about the 'DX(Developer Experience)'.

‍

Have you heard of 'DX(Developer Experience)'?

‍

Just as UX(User Experience) is important for users, DX(Developer Experience) is important for developers. An ideal environment for developers to focus on their work is more likely to lead to increased productivity for the team and the company, which is why some companies have 'DX Engineering' teams dedicated to improving the 'Developer Experience'. DX encompasses many different categories, but the main one is 'Development Environment Infrastructure Improvement', which includes 'CICD', 'Test Automation', 'Monitoring', 'Notification System', etc.

Now, Ntuple has a good 'Branch Strategy', 'stable deployment process', 'CICD', 'test automation', 'monitoring', 'notification system', etc. Of course, we're still finding improvements as we speak (in fact, there was a small uproar about the development environment recently...😅).

Today, I'd like to briefly introduce some of the things we've tried to improve Ntuple DX, hoping that this will give you a little hint of how to improve your development environment if you've been feeling uncomfortable with it, or give you a chance to take a quick look at yours if you already have an optimal development environment.😊

‍

Check out your current development environment

‍

First, think about your current development environment. I've experienced roughly the following five environments.

✅ Individuals are given their own remote development servers.

✅ You're given different users, directories, and subdomains on a single development server, and you register with a host to develop.

✅ Develop in a container environment such as Docker on your personal PC.

✅ Develop locally as a Thing, not in a container environment.

✅ Run VMs on your own PC to develop.

What about you? How about taking a look at your current environment with the checklist below?

✅ Are you happy with your current environment?

✅ How is the development environment being prepared for new members?

👉 Your team mate or infrastructure team has created it.

👉 There is a manual.

👉 It's told like a folk tale.

✅ Are there any "collaboration" issues?

👉 My environment and the other member's environment are very different, and I'm doing what I'm clearly told, but it's not working.

👉 I spend a lot of time setting up preferences for each task.

👉 I want to reproduce a behavior in a code review, but switching environments is difficult.

✅ Is it easy to 'reset the environment'?

👉 I can tear down my development environment and rebuild it as quickly as I want.

👉 Even if there is a problem with my development environment, I feel like I'm crossing a river of no return if I reset my development environment even if there's something wrong with it, so I force myself to use it.

‍

Have you ever wondered if there's a better way?

‍

What if you think, 'Improving my development environment would make me more productive overall...' but you're too stressed for time or not authorized to do it? What's your willingness to improve? Find out with this checklist!

✅ I've often wondered if I could make it easier to prepare the development environment when a new member joins.

✅ I've often thought that different team members have different environments, causing problems and costly communication.

✅ Multiple developers are sharing a single remote server, so when something goes wrong, everyone's work is interrupted, and productivity drops dramatically.

✅ Whenever I build a development environment, I feel frustrated when something goes wrong and I think, "Why am I the only one...🤷‍♂️

✅ You want to wipe your environment clean, but the sheer scope of the task makes you sigh.

✅ I spent my first day working from home setting up a development environment.

✅ I can't develop from home without a VPN connection.

✅ I wish I didn't have to do it all and that building a development environment was easier than it is.

How many did you check off? I've thought about every single one of them at one point or another.🙋‍♂️

‍

Don't miss the improvements visible only at the beginning of your career

‍

Obviously, there are things that are only visible to the person who just joined the team. Things like documenting the trials and tribulations of building a development environment, or improving the onboarding checklist will help the next person, so I make it a point to ask new members to complete the onboarding checklist when they join.

Here are a few improvements I noticed at the beginning of my time at Ntuple.

‍

1. Build an environment, manage as 'sources' rather than 'documents'

2. Collaborate on multiple tasks in 'parallel'

3. Introducing 'Docker Compose'

4. Adapt and modify schemas flexibly and independently

5. Include multiple configuration files in a history management target

6. Keep your most frequently used commands in one place and manage them as code

‍

Here are the top 3 improvements we've made based on the above.

‍

1. git

2. Docker Compose

3. Makefile

Let's take a closer look at each one.

‍

1. Create a 'Git' repository for history management

‍

The benefits of managing with code include

‍

✅ Manage configuration history

✅ Separate implementation branches for each requirement to work independently

✅ Everyone can participate in improving the environment

✅ Easy to recover

‍

I wrote how to set it up in the README.md to create a single entry point and make it convenient to share only the repository.

‍

2. Let's use Docker Compose

‍

If your service is ready to be built as an image via Dockerfile, you can easily adopt Docker Compose. What are the benefits of using Docker Compose?

✅ Easily manage and deploy multiple container configurations for development via a compose.yaml file

✅ You can override multiple YAML files, so you can split them into separate files to manage "per-environment" and "per-feature" as needed.

✅ If you have compose.yaml created by someone, you can build an environment with docker compose up -d command without long docker run command.

✅ Manage your own environment builds

‍

If you have written a compose.yaml file, you can spin up your environment with 'docker compose up -d' and stop the spun up services at once with 'docker compose down'.

‍

There are many other directives and features, so feel free to play around with them.

‍

2-1. Now let's run DB, Redis, and everything in a container!

‍

Initially, a small number of developers shared a single development server DB, so we can speculate that it was maintained for the following reasons.

1. The number of developers was small, so there wouldn't have been many parallel developments going on, and they could have been resolved to some extent through communication.

2. The DBA team was managing the development DB and would have requested and used schema enforcement when needed.

However, as the team expanded, various features started to be implemented more and more frequently in parallel, and it seemed that using one DB for various requirements could be a delaying factor. Also, since the DBA team was managing the schema directly at that time, it was difficult for developers to modify the schema and develop at the same time due to the nature of development where development data is frequently tested.

So, here's what we did

Configured the DBs to be spun up locally on their own with Docker Composer

⏬

Requested a full schema from the DB team

⏬

Include a script to apply that schema to the local DBs

At first, it was pretty simple. Every schema change was a request to the DB team to overwrite a file, which I did manually for a while because the changes were infrequent and not a difficult task. Then, as the DB team also maintains schemas in git, Bradley improved it by automating it with a git submodule to make it easy to apply the schema!

Dev teams are now free to build and use their own DBs, independent of the state of the remote development DB. The benefits of this approach include

✅ Quick testing before the DB team schema is finalized.

✅ Business continuity was secured as development could be done anytime and anywhere, unlike before when problems with remote DB servers affected development.

✅ Separate branches for each agenda facilitated parallel development.

Afterward, Redis, which was included in the source image, was also separated into a separate image, allowing us to respond flexibly to Redis version upgrades.

Let's automate environment deployment with a Makefile

‍

Now we've created the repository, environment build with Docker Compose can be managed as a source, and we have our own DB in a separate space!

However, to actually develop and maintain the environment from the environment building stage, I needed the following various commands.

🧾 'docker compose up -d' to start the environment

🧾 git-related commands to clone, fetch, and pull sources for all services

🧾 Composer, npm, and other commands to manage dependencies

🧾 Eliminate the need to DM commands for applying DB schema, or for different developers to run them differently and get different results.

So I created a Makefile to bundle the necessary processing and make it simple to execute.

Makefile is a tool that facilitates compilation in Linux environments. It has many features, but one of the best is that it runs step-by-step, which allows you to bundle multiple processes together and automate them.

You can do this as a shell script, but I personally find it easier to use and manage.

For example, let's say you write a Makefile like the one below. When you run the 'make local' command, it will go through 'start container', 'clone source', 'build', and finally, it will automatically open the local host in a browser, making it easy to automate environment deployment.

This is very useful because the commands you define in a Makefile can be used in other commands, and you can take advantage of features already available in Makefiles, such as shell scripts, variable handling, and macro settings.

‍

After the change...?

‍

At first, there were a lot of things that didn't work as well as we thought they would, and there were a lot of questions about why we needed to change the environment we were already using, but we've been improving it little by little, and now our developers have a common environment to develop in.

We still have a lot of work to do. The services are getting more complex, and there are more environments to check. There were times when we introduced things to make development easier, but it became a liability because of the increasing complexity. But I think it's good that there's still a lot of things we can do!

At Ntuple, we're doing a lot of things to help new members get settled into the team and the work, including 'onboarding', efforts for a better development environment, development culture, and processes, and 'branch operation strategy', 'deployment strategy', 'test code automation', 'QA automation', 'CICD', and 'laC' for stable service deployment... We believe that these things can eventually create a 'good developer experience'.😊