Why I love agile scrum for data science teams
At my last company, I helped implement a version of agile project management on our data science team and ended up really loving it. In this post, I want to share a bit about how we implemented agile and why I think it’s such a great tool for growing data science teams at early stage startups.
Agile is a project management approach that’s very popular among software engineering teams. If you’re not familiar with it, here is one of my favorite blog posts describing its implementation on data science teams.
We implemented a very lightweight version of scrum, with these main components:
Sprint planning: a meeting with the whole team to figure out what you’re going to work on for the next two weeks.
Regular standups: a meeting or slack update to share what you’re working on today, and quickly surface any potential blockers.
Our async standups were a fan favorite: we had a slack channel where folks shared what they did yesterday, what they were doing today, any blockers, and how they were feeling. It was especially fun to scroll through the channel on Mondays and Fridays as people shared about their weekend plans and how they were feeling about the week.
Demos: a show-and-tell meeting where the team shares what they did that sprint. Our demos were open for anyone at the company to join.
Retrospectives (retros): a productive vent session for team members only, where we talked about how the two weeks went. We had a simple Google jamboard where we placed sticky notes in “Good,” “Meh,” and “Bad” columns.
Up until we were about 10 people, we did everything all together. Once we had more specialized sub-teams, we started to split meetings up. Standup was first to split, with separate meetings for each subteam. Then we implemented breakout rooms during sprint planning, where each sub-team would plan on their own and then come back together to share their sprint goals with the whole team. We kept demo and retros as teamwide events as we grew. I’m sure at some point these need to be split up, but at 20 they were still very manageable, though we did have to ask for sign-ups and manage time and side-conversations more carefully during demos.
The biggest data science-specific limitation we discovered was that backlog tasks actually weren’t great for tracking those pie-in-the-sky, “maybe one day” R&D projects that we didn’t have time to do right now but didn’t want to forget about. We ended up creating an “icebox” in our wiki to throw those ideas into, and that worked decently enough.
Benefits of implementing agile scrum
As we developed our agile practices, I realized that it came with a lot of benefits. Agile:
Minimizes the emotional burden of deciding what to work on. Because you’re isolating all the mental overhead of deciding what you’re working on to just one day every two weeks, you can spend the rest of your time heads down working. Before implementing agile, I found myself going through the emotionally draining task of deciding what to work on next after each task I finished. But once I started to plan for two weeks of work up-front, it was easy to decide what to do next because there were very few things to choose from. Of course, there’s always some wiggle room with tasks getting slurped into a sprint that weren’t initially planned for, but in general it removes a lot of the tumultuousness of constantly re-shuffling the tasks you’re supposed to be working on.
Forces you to break down complex projects into smaller tasks. A common pushback from R&D scientists is that you can’t do a big data science project in two weeks, or you can’t predict how long something will take. Sure, but then go ahead and write yourself a task that reflects the amount of work you can do in two weeks. Get creative, and besides it’s just guidance not strict rules. (This post gives nice examples of data science deliverables).
Helps you say no and it also helps you say yes. If someone asks you to do something in the middle of the sprint, it’s easy to say no, your sprint is full for now - but also yes, you’re happy to revisit their request at the next sprint planning.
Helps other teams understand why you’re saying yes or no. Because there’s a principled way to prioritize between tasks and all tasks go through the same prioritization process, it makes the tradeoffs between requests visible and explicit. If another team asks you to do something that would require de-prioritizing some critical work, you can show the tradeoff you’d need to make to say yes. Most times, they’ll understand – they don’t want to jeopardize some greater company goal. But if they don’t, it still makes for a more productive conversation: the tradeoffs are explicit, and if you need to escalate the prioritization decision it makes that a much clearer conversation.
I also found benefits for my situation specifically as an early employee wearing multiple hats in our growing team:
As a manager, it helped me keep tabs on my direct reports without micromanaging them, and was especially helpful when I was very busy. I could get updates on their work passively as they updated their tickets without needing to ask for daily status updates or to find time to do in-depth code reviews. And if I saw them going down an unnecessary rabbit hole, I could just leave a comment on the ticket to hold off on doing any more work until we had a chance to connect.
Also, I was able to identify when my direct reports were blocked, especially junior folks. If I had estimated a ticket to take just a few days and they were stuck on it for a whole week, then I knew to check in on them and make sure they weren't missing something that was otherwise obvious to me. Or if I saw their “ask devops for access to X” ticket languishing over multiple sprints, I knew to ping the devops manager to see what was going on.
It made our work so searchable. We had really good ticket hygiene: we commented on our tickets with status updates and included links to where we’d done or documented the work. So if I was looking for a project that someone had worked on six months ago but had no idea where to find the code or write-up, I could look at the ticket and usually find some useful bread crumbs.
Demos and retros
My absolute favorite part of agile was the demos and retros. I found them to be crucial to building remote team culture.
Demos helped us create a culture of sharing in progress work, celebrating each other’s wins, welcoming other teams, and embracing curiosity and open discussions. Retros were also great, but for slightly different reasons. Rather than focusing on what our team did, retros are the time to talk about how we did it – essentially a vent session. But because it’s a vent session in the context of work, they necessarily become productive vent sessions. I found that retros were so helpful for building a team culture that was empathetic, supportive, and - importantly - empowered to improve their own situations.
I love demos for so many reasons:
They give junior folks a way to contribute to team discussions. Because you’re presenting on the sprint work you completed, everyone can contribute. It doesn’t matter how big or small your ticket was, if you did something then you should show it off!
They give the team visibility into the progress of larger projects. Some R&D projects take months of wandering before they turn into anything concrete. At demo, folks working on those projects can present on the random paths they took this sprint. This is especially helpful for senior scientists who work very independently: junior folks get visibility into what it actually takes to complete a large R&D project, team leads get visibility into what their senior folks are actually doing, and senior folks get to be consistent contributors to the team culture.
Presenting less stuff more often also means that whenever the project does wrap up, the rest of the team is already primed on context and will be better able to engage with the final presentation of the work.
Demos are a great place to share miscellaneous work that is valuable but doesn’t really fit anywhere else. Because there’s no expectation of “wow” factor, you can share literally anything at demo. Maybe a weird result during a larger project made you take a detour into the depths of your company’s data model, and you want to report back what you discovered. Or you spent a whole week fighting AWS Batch and you want to show off what you learned so no one has to struggle through what you did (or you just want others to know how much pain you were in). Or you were bored one day and randomly did a small analysis, and it’s not earth shattering but you found it fun and interesting - now there’s a place for that analysis to go!
Because anybody at the company is invited, demos are a place to forge interesting alliances. Maybe the data team has been struggling to connect with your Growth team, but they just hired someone who’s curious and starts to join demos whenever she can. That’s an ally! Or maybe your design team is underwater and can’t spend much time working with your data scientists on improving their data visualizations, but they can attend demo and give small bits of feedback there. Easy win!
For me though, retros were the most game-changing meeting we implemented. When we implemented agile, I was still the only manager on our team of five which was an extremely lonely role. Implementing retros gave me a way to democratize my leadership burden and an appropriate place to connect and vent with my colleagues about work.
I think that retros benefited everyone on the team:
Feel validated in your pain. It’s not just that you’re stupid and bad at coding, it’s that everyone struggles to install Python packages their first time. And no, you didn’t miss something obvious - everyone struggles with access to the data oh and by the way here are some tips the team has built up to smooth the bumps.
Inject positivity into bad weeks. Some weeks, I found that all of my comments were in the “bad” column. Seeing my colleagues put things in the “good” section made me rethink the sprint – oh yeah, that thing that happened was good and oh, here’s something I forgot that was actually really nice.
Talk about how personal or world events are affecting your work. On a really bad news week, it can help to know that the whole team is struggling even if no one has brought it up in other contexts. (This was especially powerful during Covid, and during the winter weeks of daycare chaos that continue to plague working parents). And if you’re having a rough week in your own life it can be cathartic to just throw it up on the board to vent and feel seen.
As a team lead, I found retros to be especially helpful as I and other data science leadership managed growing our team in our startup environment.
Surface team-wide pain points that could be addressed through technological or organizational improvements. We often saw patterns in what people were bringing up, which helped us identify growing pains that we needed to address. For example, when we saw folks consistently getting inundated and sidetracked by ad hoc requests via DMs, we realized we needed a ticketing system to support our team. Or sometimes we’d see that everyone was having trouble accessing a certain database or understanding the data in it, which meant that we needed to go connect with engineering and sort things out.
Keep a pulse on team morale. This one was huge - retros are really where you get a sense of the vibe of the team overall. As a team member, this is great for solidarity but as manager it’s so critical to get a pulse on how things are going in general. Especially in a growing startup, the company will have rough weeks - retros are where you can see how deeply the roller coaster is affecting your team.
Surface existential questions that you need to get answers to. Sometimes the vibe was off because there was an existential question about the company hanging over the team’s heads. Retro was where those patterns became apparent - it wasn’t just one or two people wondering how long our runway was and if they needed to be looking for a job, it was the whole team who was worried. As a manager, being able to see these patterns and show them to our leadership was extremely helpful in getting answers.
Rehash big company events, and maybe even inject your own take on them. As a safe and closed space, retro is also a place where the team can reflect on company announcements or milestones. As a leader, this can also be a place to provide your perspective or additional context that can help folks understand why certain things are happening.
Create a culture of empowered problem-solving. Instead of having your direct reports complain to you 1:1 and piling the responsibility of solving their problems solely on you, you can ask them to bring problems up in retro. There, you can get a sense of how many other people are affected and also ask the team to brainstorm how they’d like to see the problem fixed. In the case where the problems are fixable, that creates a culture of empowerment - your team is involved and bought in to the solutioning process. In the case where problems aren’t fixable, they start to understand that there are no magic fixes and you can direct the conversation to more productive ends (like where their sphere of influence actually extends to). This was by far my favorite byproduct of retros.
Get help fixing the team’s problems. In certain cases, I found that I could delegate some of the problem-solving follow ups to the team members who had brought it up, were most passionate about it, or who had the most relevant relationships to get things moving. As the only manager on the team, having a way to facilitate getting help solving the team’s problems lifted a huge weight off my shoulders.
I know there's a lot of content out there about agile and implementing it on a data science team. I hope this perspective helps break through some of the nitty gritty implementation posts to help you think about why it might have a beneficial impact on your non-software team. As always, if you’re interested in chatting more about implementing these or other concepts I write about at your early-stage startup, please don’t hesitate to reach out!