Coronavirus has affected every walk of life and the tech industry is no exception. The coronavirus outbreak has forced people to work from home. The offices that used to be filled with chatter about why tabs are better than spaces are empty today. Companies have moved from being office first to remote first. This change in office dynamics necessitates changes in how we work and how we communicate. This blog post is about the lessons I learned working remotely.
At Dgraph Labs, we’re building the world’s most advanced graph database and it comes with its own set of challenges. The challenges are not limited to technical but also require communication and collaboration skills as well. Proper tools for communication and collaboration are needed as much as a powerful computer. When building a complex, performance-critical system, it needs to be benchmarked and tested under heavy load. Every aspect of the system has to be investigated for memory leaks and performance bottlenecks. So, every engineer gets a powerful machine, which is a core part of their daily workflow. The engineer must have access to their machine 24x7.
When you’re working on building the most advanced graph database, pushing the system to its limits to unravel bottlenecks is a daily occurrence for each engineer. For this reason, all engineers at Dgraph Labs get machines that are loaded with horsepower that includes multiple memory sticks and SSDs. We benchmark, profile, and squeeze out every bit of performance. But since most of the engineers are working from home for the past few months, it becomes necessary for them to be able to access their computer. SSH is an easy way to access your computer sitting at your office desk but it requires your computer to be visible on the internet, a.k.a., it needs a public IP. Most of the engineering team works out of a co-working space and the computers are kept inside a private network. So SSH-ing it directly is impossible but there are ways to access the computer in the private network.
Ngrok is an amazing piece of software that can be used to do this. Tailscale can also be used for the same purpose. Here’s how you can set up ngrok to access your computer
First things first, download the ngrok binary by following the Ngrok setup page. Once installed, we will need to start it so that a local ngrok process can start receiving requests from the internet and forward it to the SSH daemon. Run the following command to expose port 22 (which is the default port for SSH)
ngrok tcp 22
Once ngrok has started, head over to https://dashboard.ngrok.com/status/tunnels. The page lists all the tunnels that are running. It should contain the hostname and the port you’ll need to connect to your computer.
The picture above shows that 0.tcp.in.ngrok.io is the hostname and 12592 is
the port I should use to connect to my computer. So let’s connect using this
Hostname:IP
ssh <username>@0.tcp.ngrok.io -p xxxxx
Ngrok can be very slow at times because it has to forward the traffic from your
computer to the ngrok server and then forward it from the ngrok server to the
host computer. But this isn’t a tough problem to solve. Luckily, ngrok has
servers in many regions and using a server located closer to your geographical
location means lower round trip time and faster connection speed. You can
change the location of ngrok server using the -region flag.
ngrok tcp -region xx 22
I am located in India and changing the ngrok server location from the US to India increased my connection speed by at least five folds. You can find the list of regions here https://ngrok.com/docs#global-locations.
Computers are interesting. They work fine all the time but it’s only when you
absolutely need them, you’ll accidentally press the power switch or run sudo shutdown -r now . On a normal day, the computer will restart, and you will get
back to work, but when you’re working remotely, you could lose access to the
computer. The only way to restore the access would be via physically accessing
the computer which may not be always possible such as during this pandemic. If
you’re using ngrok for SSH connection and the computer crashes, the ngrok
process will not start again and the process has to be
manually started through physical access. But there’s a way to fix this. Ngrok
runs as a binary and to ensure this binary runs every time the computer starts,
we can set up a service. Once this service is created and enabled, ngrok will
start automatically every time the computer starts. A service is a background
process that can be configured to start automatically. Here’s how you can do
that on Linux with systemd
Create a new service file
vi /etc/systemd/system/ngrok.service
and add the following to it
[Unit] Description=ngrok ssh tunnel
[Service] User=<your username>
ExecStart=<path to ngrok binary> -region xx tcp 22
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target
Save and exit the editor. The service we just created is ready to be started. Enable the new ngrok service so that it starts automatically
sudo systemctl enable ngrok
sudo systemctl start ngrok
That’s it. Ngrok will now automatically start every time your computer starts.
When you’re working in a team, there are times when you could use an extra pair of eyes while coding. Pair programming is a common practice at Dgraph where two engineers work together on a piece of code. When working remotely, people can share their screen and work together. Screen sharing is the old school method that will get the work done, but the problem starts when a person wants to point to some specific code and make changes. Pair programming via screen sharing is tedious and error-prone. When the other person can only see the screen, it’s tough for them to participate. Screen sharing isn’t pair programming, and there are many fantastic tools out there for pair programming.
One of them is the Visual Studio Live Share plugin. This plugin works only with VS Code but it works like a charm, For people using vim, you can do pair programming using tmux and SSH as well. I use the VS Code Live Share plugin excessively and it allows me to work effectively with other engineers.
At Dgraph Labs, we have a culture of asynchronous communication. When you’re working in a team that is distributed across the globe, synchronous communication is seldom possible. The day for half the team ends when the day for the other half begins. To bridge this gap, we use discuss.dgraph.io and encourage people to write things instead of scheduling meetings. Discuss.dgraph.io is a forum and our internal wiki platform. All the RFCs and technical/non-technical documents are published to discuss.dgraph.io. Most (if not all) decisions are made on discuss.dgraph.io. An added benefit of this writing culture is that everyone gets to participate in the conversation. Meetings could get sidetracked, one person might end up talking for too long, people could have a poor internet connection; a write up instead of a meeting saves everyone’s time and allows everyone to put forward their thoughts.
Though discuss.dgraph.io is great, it doesn’t mean all meetings should be converted into a write-up, some things can be done faster if people can get together on a call. A writing first culture helps us use meeting time more judiciously.
We use Slack for our internal communication. When used correctly, it allows us to spend less time talking and more time working. But all too often, it ends up bombarding us with red notifications. I believe in the “Don’t ask to ask, just ask” principle which means that if you want to ask a question, just ask the question right away.
Foobar: Hi Ibrahim. I have a question.
The message above doesn’t provide me with any context except the fact that Foobar has a question and wants my attention. This kind of communication demands synchronous communication. A better way of doing this would be to ask it as
Foobar: Hi Ibrahim, I’m looking into Badger and trying to understand it.
Can you please answer the following questions
<some questions>
This is an excellent way of asking questions. I have all the context I need to help Foobar, and I can help him right away.
Working from home is tough but it doesn’t have to be. Having fixed work timings, setting up a proper office, ensuring you have a good internet and audio/video setup might sound like simple things, but they can make a huge difference. The office chatter is irreplaceable, but watercooler talks and Friday games over video conferencing are what keeps me sane. Working from home for the past few months has taught me numerous lessons, and I hope the tips and tricks I’ve mentioned here will be helpful to you as well.