Huge speed improvements for 8 slow steps

Announcements
1

Hey all!

We’ve significantly upgraded 8 of the most used, and sometimes slowest, transform steps.

Most should run about 15x faster than they used to, which means if a step used to take 1 minute to finish, it will now only take 4 seconds. :rocket:

Some steps will already be faster in the flows where they currently exist, and some will need to be replaced to get the speed boost.

Steps that will already be about 15x faster in existing flows:

  • Merge duplicate rows
  • Remove duplicate rows
  • Insert Row Numbers

Steps that need to be replaced in your flows to get the 15x speed boost:

  • Count by group
  • Sum by group
  • Average by group

Steps that will already be faster, but only about 4x faster :grinning_face_with_smiling_eyes::

  • Filter rows
  • Insert if/else column

These steps were optimized for working with non-numbers. If you are using these steps to work on columns of purely numbers already, they will not seem much difference, but they should be fast already!

Hope this help you all build faster and better.

-Brian

5 Likes
2

Awesome!

Any way to tell if a step needs to be replaced or if it’s already faster?

And would duplicating (and then replacing) an existing step work to get the speed boost?

3

Hey Brian,

If it is a Count by group, Sum by group, or Average by Group, and the grouping columns that you are selecting are not purely numbers, then you should replace them!

The other steps should all be faster already, unless you have a very old version of those steps still active.

Duplicating will not work, as that duplicates the step’s version as well. You’ll need to pull a new step off of the sidebar and configure it. Unfortunately that will be a bit slower than just duplicating, but hopefully the speed gains make up for that!