Happiness is… a freshly organized codebase

Starting From the Top

Imagine joining a new team at a new company and you finally get access to a codebase.

Where do you add a source file for your team’s project? Great question. Your new team has three source directories for you to add files to, and oh, you’ll have to figure out which one you want to use, which one your teammate uses, and where you’re going to find the files that need refactoring.

The Slack iOS team lived in these conditions for a few too many years. We got here as a result of some attempts to organize source files (several times), a lack of architecture pattern in the codebase, and a high growth of developers over a couple years. To put things into context, we have roughly 13,000 files (and counting), about 27 top level directories, a mix of files in Objective-C and Swift, and around 40 iOS developers that work in one monorepo.

These are all real photos of the state of our file hierarchy. New hires constantly expressed frustration jumping into the codebase — and rightly so, since some of us just got used to navigating these chaotic directories and didn’t remember the pain of starting from scratch. Not only did we have numerous source code directories (our prized iOS, SlackCocoaSDK, and Slack directories), but it took a lot of time to settle on a directory and then to decide how to add a file. On top of this, we decided we wanted to add new tools to our codebase and the current state of our Xcode project did not lend well to supporting them.

As a result, a group of iOS developers and I set out on a mission:

1. Make it quick and easy to add a new file — for any developer new or tenured
2. Follow our design pattern in directories
3. Enable ourselves to maintain a new and cleaner folder hierarchy with the use of tooling

We took on these goals in two stages: move high level directories into a coherent order (things like main target directories, extension directories, frameworks etc) and then — the bigger task — source folder organization.

The top level directory moves were neither contentious nor hard. It probably took us only a few weeks with a few developers. In these first moves, we learned a few things to take into our following stage — tackling big moves during off peak work hours, consistently merging master (who wants merge conflicts?), and working with folks on timely reviews. Merge conflicts weren’t the only struggle we’d encounter during this process, but we could have invested in a better way to mitigate merge conflicts with xcodegen since most conflicts were in the project file. We also wanted to think about preserving our git history and maintaining a clear contract with how we see files displayed in git and finder. However, we opted for easy enablement of moving files to get folks involved and dragged and dropped files into their new homes.

If you look at this image from September 2018 you can see we were able to organize our top level fairly successfully. A place for every directory and every directory in its top level place, right?

Source File Organization

Time to tackle the source files in iOS, SlackCocoaSDK, and Slack to move them all into App/Source. Honestly, I dreaded this part. We needed consensus on a pattern, a clear way for developers on all teams to get involved, and both rules and tools to make sure moves were easy and clear to engineers if they had done something wrong. I did a lot of investigation into patterns for our hierarchy and there’s a surprising lack of articles on folder organization. I had thought this would be a well documented topic going into this journey, but as a less glamorous part of the job, I can understand why it isn’t covered. Of the articles I did find, Uber wrote this article about how they approached the move to their monorepo. This gave me some insight into how we could chunk up our codebase into modules (but on a smaller scale).

I ultimately presented three options to my greater team: Feature organization, Topic (architecture based organization), and Ontology (relationship or similar grouping based organization). The group converged on Feature for top level and then Topic or MVVM+C organization inside of those feature directories.

Here’s an example Folder in our new structure:

/FeatureFolder
   /Coordinators
   /Models
   /Tests
      /Functional
      /Mocks
      /Unit
   /ViewModels
   /Views

Moving the source files proved tedious and cumbersome. Merge conflicts were annoying, small things like searching for a file namespace to see if you captured all the feature files into a directory proved harder to remember than originally thought, and lots of files we were moving fell outside of the folder rules we set out initially. Thankfully, we had a few heroes step in and make some giant moves to remove iOS, SlackCocoaSDK, and Slack all into App/Source.

A snapshot of our folder hierarchy from January 2020.

While we were moving these three large source directories, we initiated a Danger rule to ensure folks would stop adding files to these directories and start using our new pattern. Danger is a tool we integrated into our Continuous Integration system that performs post-commit automated checks and posts warnings and errors onto PRs. This is what one of ours looked like:

has_slack_directory_additions = !git.added_files.grep(/Slack/).empty?has_slackcocoasdk_directory_additions = !git.added_files.grep(/SlackCocoaSDK/).empty?has_ios_directory_additions = !git.added_files.grep(/iOS/).empty?if has_slack_directory_additions || has_slackcocoasdk_directory_additions || has_ios_directory_additions   fail(‘This PR is introducing new files into directories that are
   closed for adding new files. Please add files to App/Source using
   the new convention found in <ahref=”…Adding-a-file-to-Slack-  
   iOS…”>Adding a file to Slack iOS</a>’, sticky: false)end

Follow The Linter

You might think “Wow, that looks great, you must be done!”. Well, not entirely. We’re still working on moving around what’s within the new source directory, App/Source. Here are some of the rules we started to implement in our “folder housekeeping”:

1. No more spaces in folder names (those tools we want to add, like Bazel, don’t play nicely)
2. No more dumping ground folders like “Helper” or “Utility”
3. Co-locating tests (if we want to really emphasize discoverability in our new hierarchy, we need to step up this game with tests and what easier way to find a test than if it’s located in the same feature directory with the source code)
4. Sort your files and folders! (who doesn’t love some alphabetical ordering?!)
5. Make sure your files live in a folder that is associated with the proper top level directory for their target

One rule that really needed tooling was co-locating tests — for this we opted for a Danger rule. Any new PR with new files added could not be added to App/Tests.

This looks something like -

has_app_test_directory_additions = !git.added_files.grep(/Test/).empty?if has_app_test_directory_additions   fail(‘This PR is introducing new files into directories that are
   closed for adding new files. Please add co-locate tests using the
   new convention found in <a href=”link-to-our-housekeeping-
   doc”>iOS Folder Housekeeping Checklist</a> or feel free to ask in 
   #ios-testing-folder-structure’, sticky: false)end

The folder committee created a Slack channel where folks can ask questions if they aren’t sure where to add a file. Confusion around where to add a file happens more than you would think and even the smallest moves can have the strongest opinions attached to them.

We’ve made a lot of progress, gotten a lot of support from the greater iOS team, but still have more to do. This is not a one-person-job, you need to enlist folks who work around all parts of the codebase. You need the greater team not just for support in moving files, but for modifying rules and adding more tooling. Having the extra hands means that a lot of folks will learn the process and be able to show newer folks how adding a file in our codebase works.

The recipe for success and happiness is surprisingly short: If you are like us and have a monorepo, create a committee of folks dedicated to the process, and set hard and fast rules. Rules can be broken, but with any number of developers you may have discussions and will need a core group to create tooling to support rules for file organization to become more natural. A core committee can also spend time thinking and researching the best file structure that works for your team organization or working patterns.

It makes a world of a difference when developers understand where to add files and can do so in a way that promotes speedier development and genuinely makes it more pleasant to be inside the codebase. Tools like SwiftLint, Danger, or homegrown scripts are great for initiatives like these. However, a big caveat is that you’ll first have to get to a point where tools become useful, which usually requires non-trivial manual labor. In short: Use tools, get folks involved, and tackle it like any other project that matters for your company. It’s an undertaking, but making it easier for everyone to find or add a file, allowing developers to understand our architecture pattern, and enabling tooling to enforce and promote a cleaner codebase is definitely worthwhile.