With the recent rollout of NewsCred’s Integrated Marketing Edition, we’ve updated our CMP with various new functionalities. While our users are thrilled by these upgraded features, we know that some of the additions have the potential to become performance concerns. Because speed is a key part of user delight in software, we set out to find a frontend performance-tracking software that can shed light on what our users are experiencing. Our goal is to be able to track each feature’s load time, identify bottlenecks, and diagnose performance problems before they affect our users.
Choosing the right software
The two products under evaluation are New Relic and AppDynamics. Before committing to one, we want to make sure it has all the features we knew we need, as well as the ones we might find essential as we continue to fine-tune our app’s performance. We’ve broken down the most important components and measured how each product laddered up to meet our needs.
How they compare
Initial load time
How long our whole application load takes and what percentage of that time is spent on API calls vs render.
New Relic: Displays the load time trends broken down into FE and BE. This view is fairly opaque about what constitutes FE and BE but you can always make a custom dashboard to drill into the details.
AppDynamics: Displays this data in a table view where you can look at all of your page load events and the timing associated. From there, it is easy to drill in and see a timeline representation of load events.
Route change load time
How long our application load takes when moving between pages within it.
New Relic: Displays the route change load time broken into JSduration and waiting time. As with the above, we found it more useful to make a custom dashboard to drill into the data.
AppDynamics: Displays route change data in the same way it displays initial load data.
Winner: AppDynamics (though in both platforms, route change data shows a less granular view of what makes up load times versus initial loads).
Interactions that happen after a page has loaded. (Ex: A user searches through a paginated list of tasks. We want to be able to track how long the user is waiting for those results to return from the server and render.)
New Relic: With the Browser Interaction API, it is easy to track inline user interactions outside of page loads and add any data you might want to filter the interactions by.
AppDynamics: With markVirtualPageBegin, it is easy to track inline user interactions outside of page loads, but it is not possible to add custom data to the event, if you name them uniquely, you’ll hit the limit.
The server calls that Single Page Applications make as users interact.
New Relic: Shows the most time-consuming AJAX calls broken down by throughput, response codes, data transfer, and other details. This view is more limited if you do not also have their backend tracking software.
AppDynamics: The view for AJAX timing is similar to the views for route changes and initial loads. You can take a look at overall timing from the high-level table view and then drill into longterm timing in the individual view.
Ability to build out dashboards to showcase the data for specific teams/use cases.
New Relic: Has NRQL, which is their SQL. It’s very easy to use, especially if you have experience with SQL, and we found the dashboards and data apps easy to create, copy, and export.
SELECT percentile(duration, 90) FROM BrowserInteraction WHERE browserInteractionName = <interaction name> TIMESERIES SINCE 2 weeks ago
AppDynamics: Has their own query language that is very comparable to New Relic’s. On top of this, they have the option to create drag-and-drop style charts, without needing to use the query language.
How long will they keep your data?
New Relic: Allows you to pay for retention, so we were able to choose the duration that felt valuable to us (1 year).
AppDynamics: Has a 90-day data retention policy, making it hard to see how a feature has improved over time.
Winner: New Relic, but only if you’re interested in long-term trends.
Can you find answers in the forums and docs?
New Relic: Troubleshooting on New Relic is easy thanks to their community forums and detailed documentation.
AppDynamics: We rarely found the answers to our AppDynamics questions in their documentation or in forums.
Winner: New Relic (though for both platforms, we found surprisingly little about setting up inline events).
Data volume limitations
Platform limitations on how many pages you can track or how many events it will capture.
New Relic: Does not have limits on the number of browser interactions it tracks. Instead, you pay for larger volumes of data.
AppDynamics: Has a hard limit of 500 on the initial load/route change names it tacks. This includes any custom virtual pages that you might want to implement (described above in inline events). There is also a separate limit of 500 for AJAX calls. If we filter out any 3rd party API calls and segment our own API calls in our configuration, this could really be an issue for us.
Winner: New Relic
Ease of configuration
Whether the software is plug-and-play or needs configuration to track effectively.
New Relic: Picks up the IDs in our routes and aggregates them where appropriate. /tasks/:id turns into /tasks/*. This process is imperfect when it comes to AJAX calls and most of our own API endpoints are currently being grouped into /api/*.
AppDynamics: Does not do default ID aggregation. By default, it will name by the first URL segment. /tasks/:id and /tasks/:id/copy will both become /tasks.
*Given our many routes, the best solution was to create a list of 12 REGEX expressions to locate and remove ids from the URL to create the name. This was successful but would require maintenance if we change our route structure. If the REGEX missed and a page was given an individual name for each id, we would quickly hit the limit described above.
Winner: New Relic
Who ultimately won?
AppDynamics displays its data well, helping the user immediately understand the state of their application. Beyond that, it is very easy to drill deeper into the details and troubleshoot the underlying causes of slow loads. However, the configuration needed to set up our data properly on AppDynamics, combined with the data volume limits, outweigh its advantages.
On New Relic, all of the same data is at our fingertips. Plus, we are able to leverage their custom dashboarding to get the most out of it. As a bonus, the custom dashboarding is easy to pick up and is familiar to anyone with SQL experience.
After testing and much deliberation, we’ve chosen New Relic because it requires less configuration and ongoing maintenance. We’ll miss the great default dashboards on AppDynamics, but we’ve already set up our own in New Relic and are enjoying watching our load times go down as we work on performance.
Originally published on May 7, 2019 11:00 AM