Remixing a Symfony

[♪ Music playing ♪ Hey everyone, welcome! My name is Emily Kaufman. I'm a software engineer based out of Pittsburgh, Pennsylvania. I actually gave this talk at the- I was a backup speaker at the first Remix conference, so if you ended up watching that already this might sound pretty familiar. But my talk today is going to be about the performance journey that Harvey, the company where I work, has undergone over the past few years and how that ultimately led us to Remix.

Alright, so Harvey is a grocery delivery service, where all of our products come from local farms and producers. So it started out, I think about ten years ago, as a community supported agriculture program, if you're familiar with that. It's a CSA. Basically, you pay the farm some amount of money per year and then every week or every other week, you get a box of whatever they happen to have produced in that time. So it's a really great way to support local, support your local farms and producers. So what Harvey did is it provided a platform so you could actually customize what you were getting in your box. So up until about three years ago, two and a half years ago, that's all Harvey really did. We had a number of farms on the platform from all over and we had provided the way for the customers to come in and login, view the contents of their box, make additions if they wanted to swap out stuff and then they would wait for their delivery.

And then the pandemic hit. So, you might remember at the beginning the world was starting to close down. A lot of people in the Pittsburgh area, turned to Harvey as their main source of groceries, to avoid having to go into a grocery store. And on the other side of that, all these producers that were used to going to farmers markets, setting up booths somewhere, so that you could come and actually make purchases, they didn't really have a place to go anymore. And so they were coming on to Harvey as a producer in order to stay in business. So, as I'm sure you can imagine, we had this massive influx of both customers and producers and Harvey began to grow and evolve from this CSA program into a full grocery store.

So, of course, during any kind of large scale growth in a short amount of time like this you're going to experience some growing pains and we absolutely did. This is a Lighthouse test that I ran in late 2020, and it was just a simple content page in Harvey, which is a Symfony application, and we got this performance score. This was the JavaScript bundle, the API requests, database lookups, even with the minimal UI to render on this basic content page, we had a baseline score in the 30s. So this wasn't correct. Something wasn't adding up. This, along with some upset customers, some customer feedback helped to catalyze this renewed interest and commitment to performance at Harvey. The catalog page, which is where you go to view all the products, had recently, I think a year prior to that, been converted from the Symfony, jQuery, Twig combination into part of a new React single page application. And this was hit the hardest. So the dump to React had addressed many UX concerns that we had. It modernized our tech stack, but it was still falling short in terms of performance. So the underlying architecture just couldn't handle the weight of all the new products. It was taking upwards of tens of seconds just to add something to your cart or remove something or do a swap.

And so many members were just dropping off the site entirely. But we have to remember that we had gone from offering maybe 30 to 40 products a couple of years ago to probably over 600 at this point. And so the page just wasn't scaling correctly. So when we think about fixes for this, our first iteration was kind of this crisis mode. Our engineering team got together and we said what can we do in the short term to fix a few of these issues.

So we spent a few hours sitting around the network panel in the performance tab and just walking through every step of page load and organizing what we saw into a few groups based on who would actually be working on them. So we had DevOps and networking. We had the back end, which would include API and the database, and then we had the front end. And from there, we took what we found and we triaged this into quick and easy wins, involved fixes, and this could be part of a future redesign.

For DevOps and networking, we didn't have to do too much here because it was mostly handled by services and tooling. But it was worth it to at least sit down and walk through it and make sure there weren't any bottlenecks. For the back end API, we had quite a few issues around image loading on the site. This is a hero banner that we have on almost all of our pages, and for some reason it was taking like seven seconds to load and seemed to be blocking first paint. So we did a bunch of work around image optimization and caching, and that took actual seconds off of our page load time. So I can't speak to everything that my back end coworkers did, but more involved fixes included updating the endpoints and database lookups to make sure that we're only querying the minimum needed, trying to avoid unnecessary and computationally expensive operations. An example of this is like anything that would have to go through a third party provider. Like if we were checking a user's credit card information for one, we might only be doing this on like one or two pages. So it shouldn't be in a top level like layout component because then it's going to be happening way more than it needs to. Overall, all of these updates came around just because we sat down as a team and we stared at the dev panel for a few hours and just identified issues.

For the front end, aka my problem, because I'm the only front end developer at Harvey, this is how long it takes to actually download the content, how big the script is, testing slower connections and all that kind of thing. We identified several low hanging fruit that we knew would be a relatively small development effort but would drastically increase the user's experience. So we started there. For one, the bundle was just so big, it was too big. There was so much code that a user had to download before even getting to interact with the page. And we used Webpacks. We used Webpacks Bundle Analyzer plugin, if you're familiar with it, and this helped us identify many problem areas that we were able to tackle. One of these being our use of Moment.js localization packages. So we're mostly in the Pittsburgh area. We have farms all over, but mostly at least in the United States. So there were many localization packages that weren't really relevant to us at this time, so we removed those, setting the goal that we would eventually switch to using something else other than Moment.

I went through every dependency in the package.json and just tried to figure out why it was there, if it was still needed, if it could be updated. And we were able to remove a good chunk from there.

So I have this fun, anxiety-inducing screenshot of the performance tab. This is our catalogue page. So one of the biggest problems that we had in the frontend, especially on the catalogue page, was just the sheer number of elements that we had. So I said before, at one point, maybe we had 40 product cards, each had an image and a few buttons, you know, normal ecommerce stuff. And now we have almost 600. So even without all the optimizations we're making now, two years ago, this page was fully functional, it was working fine. Because remember, Harvey never was meant to be bigger than a small CSA program. So one of the first things we did here, after seeing this, was to add lazy loading to the page, and it immediately stopped crashing, which makes sense. But you can see in the screenshot here, we're basically trying to load 500 images at once.

OK, so I won't dive too deeply into everything we did on the front end, because I only have 20 minutes, but I have a non-exhaustive list here. So better code splitting. This could even be more thoughtful code splitting. Just really taking the time to think about what needs to actually be loaded on every route. So with our single-page application, we didn't put a lot of time into considering which modules actually needed to be on each page. So one of the things this led to was we split our member and our admin code into two separate projects with their own builds, and then I made a components library where the two could share from it. So that was able to reduce our bundle size by a lot. We started using text compression. We added this to all of our JavaScript files. And I think our CSS using Gzip, and that had a positive effect. Minimize and defer third-party scripts. Sometimes this can feel like a constant battle with marketing. This is stuff like Zendesk, Hotjar, Analytics, anything that, you know, maybe marketing is using to track events on your website. And so we went through these and just made sure that all the ones that we were using were actually still needed, and tried to identify places that we could defer them where we could. Minimize expensive API calls. I kind of touched on this before, but a lot of this was just moving where we were making these calls, like out of a layout and into a component or into a route level. Just so that they weren't all being made on every page. So all these changes that I mentioned led to a substantial difference in terms of page load, and especially on mobile. And they helped somewhat with the page interactions after loading, like adding, removing, swapping card items.

But we knew that our current design just wasn't going to support the new business model, the bigger grocery store model that we were trying to achieve. And so we knew it was time for a redesign.

One thing we did, this is our redesigned page here. Instead of loading every single product on the page, we loaded them by category. So we just weren't having as many elements on the page anymore. Fun note on this, though. Our categories are growing big enough now that we're starting to look into pagination or something like that, because of the growth.

We removed parts of the product card, the description, the producer information, and we moved these into a details page so that we weren't trying to load all of those for every product in the catalog. We generated multiple image sizes for each of these, so we had a thumbnail, which you can see at the bottom of this image here. We had a details view, which is this bigger one, the card view, and they matched how they'd actually be displayed on the page. We dropped some functionality, like the swap button. We used to have it so if you were in your shopping cart, you could swap out. Like, say you didn't want broccoli, you could swap it out for carrots. But it just wasn't adding enough value for the customer for the performance issue that it was causing, so we decided to get rid of it.

And that kind of led to this way of thinking. Like, when you include a commitment to performance as an essential tenet of design, so when you're thinking about performance every time you're building a new feature, the design's going to become leaner. So I find myself doing this a lot. So we'll be like, oh, we should add this to the site. This would do this. It would be really cool. And then we realize there's going to be a performance impact. And it's like, is it really worth it to add this much time onto page loads? Is it worth it to take that hit? And a lot of times the answer's no.

Okay, let's see if this loads. So our redesign page is going to be on the left and the existing page is on the right. The redesign page you can see is loading substantially faster than the existing page. But it's still not enough. I wanted to get rid of all the spinners. And despite the fact that our load time was so much better, our lighthouse score was still fairly low, which was surprising.

All right, into a remix. In early 2020, I had bought myself a remix license as a birthday present to myself.

And I've been messing around with it on several hobby projects, just testing out what it could do and just in general having a good time. It was for myself. And I realized that the co-location of page load data requests with the component, with the layout, might help us break out of this spin again problem that we're having, as the remix team calls it. So I don't have anything deployed publicly yet, as much as I want to, but I was able to get our new grocery pages up and running with remix. And I think the results speak for themselves.

I ran this lighthouse report just to get an idea of if, you know, it's going to make any difference for us. And yeah, you can see. So one of remix's selling points is that it minimizes the request waterfall. And that was the case for us in some parts of the application. We were making multiple API requests on page load and you just had that spinner going in the UI. Let's look at this video. Is that playing? I'll do it again. So fast. There we go.

Okay, so this video is slightly skewed because the existing application has more third-party integrations like a support widget and a few more features overall. But you can still see that second wave of the requests that were fired in the use effects in the component. So that's on our existing page. If I pop over to Remix, this is the same waterfall but on the Remix version. So in the Remix version, we have a longer document load time but a minimal number of additional requests. And since we're only loading what's needed for that one page, this JavaScript that does load is a lot smaller. So this is all without taking full advantage of some of the features like nested routing which we're starting to work on now. But yeah, it was like an immediate improvement for us. It was really exciting.

So going forward, our plan is to continue to keep moving these existing pages that are currently in our member-facing code into our Remix application till we get to a point where we have enough that we can deploy it and have it be public. But using the framework has just become a natural progression of our performance journey at Harvey, and it's absolutely been worth the effort, and it helps that I really enjoy working in it. So if we've learned anything over the past two years, it's that performance and building for scale are non-negotiable when we're releasing new features. It has to be baked into everything that we do and constantly check for regression. So our product lists, our customer base, they're still growing at a rapid pace. And yeah, I feel confident that what we're doing now is going to support what the business looks like maybe like five to ten years down the road. So it's an exciting time. All right. Thank you for listening. Again, my name is Emily Kaufman. If you have any questions, you can find me on Twitter or you can read the blog post behind this presentation on my website.