r/learnjavascript • • Jul 05 '21

Remove duplicates from arrays using Set

Post image
438 Upvotes

44 comments sorted by

30

u/[deleted] Jul 05 '21

Thanks for not making this a 9 minute YouTube video with electro bass music and text on screen

12

u/zachwolf Jul 06 '21

“What’s up guys? It’s ya boi back with a crazy way to remove duplicates in an array. But before we get started, make sure to like and subscribe as well ring that bell. Doing so tells YouTube’s algorithm that this is good content and helps get this video suggested to others. My patreons got this a whole week ago, so if you like what you seem make sure to head over and subscribe. You can also join now through YouTube memberships. I’ve got new swag on my store for the true fans. If we get this video to just 5k likes I’ll do a special q&a. As always, I’m streaming on twitch Tuesday-Friday 3-7pm central. Now let’s cut right to it…

[…new Set(1,1,3,4)]

And that’s it! Again please like and Subscribe! Head on over to my Instagram if you want to see more uncut footage and daily posts.”

6

u/icy_skies Jul 06 '21

"Don't forget to join my OF"

18

u/NANOwasFound Jul 05 '21

What is the use of ... before new? Sorry if it's dumb I am very new to js

44

u/throwaway1253328 Jul 05 '21 edited Jul 05 '21

it's the spread operator

const a = [1, 2, 3, 4, 5];
const b = [6, 7, 8, 9, 10];

// [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
const combined = [...a, ...b];

// can also be used with destructuring
const [first, ...rest] = combined;

// 1
console.log(first);

// [2, 3, 4, 5, 6, 7, 8, 9, 10]
console.log(rest);

// 2, 3, 4, 5, 6, 7, 8, 9, 10
console.log(...rest);

// or with objects

const aObj = { x: 1, y: 2 };

const anotherObj = { z: 10, ...aObj };

// { x: 1, y: 2, z: 10 }
console.log(anotherObj);

7

u/NANOwasFound Jul 05 '21

Thank you : D

18

u/dvlsg Jul 05 '21

It's just turning the Set back into an Array.

const filteredArray = Array.from(new Set(array));

Would do basically the same thing.

5

u/NANOwasFound Jul 05 '21

Thank you : D

1

u/hinsxd Jul 07 '21

Not exactly. A Set is "iterable", therefore it can be spreaded to an array.

7

u/Web-devil Jul 05 '21

That’s nice to know

Problem is, we all know wtf a set it in math, but can’t remember the method lol

1

u/[deleted] Jul 05 '21

off topic, but how do you make those little code blocks

1

u/webdevolution_ Jul 05 '21

https://carbon.now.sh/ I use Carbon for it.

1

u/[deleted] Jul 05 '21

thanks

3

u/HashFap Jul 05 '21

Only works on primitive values like strings, numbers, and booleans.

14

u/revenezor Jul 06 '21

This is not true. It works on all types, including objects. Duplicates of object references are removed, and there is no such thing as duplicates of object literals as each new object is unique, even if it shares the same properties.

1

u/[deleted] Jul 06 '21

How does it consider an object when making the set? Is it if all fields are the same?

4

u/revenezor Jul 06 '21 edited Jul 06 '21

Do you mean how does it determine whether two objects are the same? The same way as the === operator. If two elements are a reference to the same object then the elements are considered “equal” and the second one is removed, or rather never added to the set.

const foo = {a: 1}
const bar = foo
const duplicates = [foo, bar]
const uniques = […new Set(duplicates)]
console.log(uniques) // [{a: 1}]

Whenever you create an object it is considered not equal !== to any previous object, even if its properties match a previous object’s.

const foo = {a: 1}
const bar = {a: 1}
const notDuplicates = [foo, bar]
const uniques = […new Set(notDuplicates)]
console.log(uniques) // [{a: 1}, {a: 1}]

0

u/FriesWithThat Jul 06 '21

If someone has a use case for it, they can create a new array with all identical values removed with something like this:

const data= [
  1,
  2,
  1,
  { id: 10, other: 'blah' },
  { id: 15, other: 'meh' },
  { id: 10, other: 'blah' },
];

const newObj = {};
const filterValues = data.filter(el => {
  if (!newObj[JSON.stringify(el)]) {
    newObj[JSON.stringify(el)] = true;
    return el;
  }
  return false;
});

console.log('filterValues:', filterValues);

3

u/nadameu Aug 01 '21

This might not work if "id" and "other" are declared in a different order when creating the object.

1

u/hpdwq Jul 05 '21

Aren't sets unordered?

6

u/Null-Fox Jul 05 '21

They use insertion order.

1

u/shiningmatcha Jul 06 '21

good to know

1

u/apple_gigworker Jul 07 '21

that sure is cleaner. though idk if i'll use it

1

u/[deleted] Jul 05 '21

In the situation you are using a strict af Typescript config, you can also do this

const array: any[] = [ 1, 5, 1, 3, 2, 1, true, false, true, 'Matt', 3, 2 ];
const filteredArray: any[] = Array.from(new Set(array));
console.log(filteredArray); // [1, 5, 3, 2, true, false, "Matt"]

7

u/[deleted] Jul 05 '21

const array: any[] = [ 1, 5, 1, 3, 2, 1, true, false, true, 'Matt', 3, 2 ];
const filteredArray: any[] = Array.from(new Set(array));

This just removes all type safety... With OP's example in TS strict mode, the resulting array is typed as (string | number | boolean)[] which is much better than any[].

2

u/Gibbo3771 Jul 05 '21

Being fair, OPs dataset is actually pretty rare to come across anyway. Normally (ideally, really) an array should contain elements that are all of the same type.

1

u/[deleted] Jul 05 '21

Either way it works with TypeScript strict mode without using an any escape hatch, though.

1

u/[deleted] Jul 05 '21

I see where you’re coming from, but you shouldn’t ever mix types within your arrays anyways for simplicity sake. The point was not the “escape hatch”, the point was Array.from. Unless you have the specific TSConfig flag, you’ll need to use Array.from over the prettier [… new Set()] syntax.

2

u/[deleted] Jul 05 '21 edited Jul 06 '21

I see what you mean now, but what you're really talking about is developing in an environment where you can not use the iteration protocol, not a "strict TypeScript config". For TypeScript this means your tsconfig targets ES5 or lower and you have the downlevelIteration flag set to false. In that case you're likely targeting IE11 with minimal/no polyfills, and Array.from won't work anyway (not supported in IE11, and since Set is only partially implemented in IE11 you would need a specific polyfill that can actually make Array.from(new Set()) work).

I see where you’re coming from, but you shouldn’t ever mix types within your arrays anyways for simplicity sake.

For unknown-length arrays containing primitives like strings and numbers, I generally agree, but there are many cases when a mixed type array is perfectly reasonable (tuples, arrays of discriminated union objects, etc). My point was that if you're using a strict TypeScript config, you should really avoid using any as much as possible.

TS can deal with a mixed type array in a type-safe way, but any just turns off type safety entirely by design.

1

u/[deleted] Jul 06 '21

Oh, damn. This is good to know. I had to turn downLevelIteration on to use the prettier method in a project I’m working on, but didn’t did too deep into why I had to in the first place. We only target modern browsers, and for whatever reason the project defaulted to false for that flag.

I had assumed it was a general default for typescript configurations.

Thank you!

1

u/20EYES Jul 05 '21

Typically if I need a unique set I just use Set in the first place. A Set can't be populated with duplicate elements so I just insert whatever comes up and if it's duplicate no worries.

What is the use case for starting with an array full of possible duplicates and then converting to a set?

I'm probably overlooking something obvious lol.

3

u/LovableBroccoli Jul 05 '21

Could be data from a 3rd party API?

1

u/JazzApple_ Oct 28 '21

Totally right that if you can just use a set from the start, but there are so many potential use cases… often relating to user input.

0

u/[deleted] Jul 05 '21

[deleted]

1

u/20EYES Jul 05 '21

As in the user provides an array?

1

u/WebDevImpasta Jul 05 '21

No, the user would just provide data or values. And then you would design the program to store the data into an array. But for whatever reason, you don't want duplicates, you can get a new array without any duplicates.

Or if you consume an API that produces duplicates.

0

u/VegetableChicken4779 Jul 05 '21

Like in Python? Awesome.

0

u/steezefries Jul 06 '21

Sets in Python are way better than JS sets

-14

u/Dazzling-Wafer Jul 05 '21

In reality, you don't have to find a 'fancy' way to remove duplicates at all. You just declare a removeDuplicates() function in the helpers folder and then reuse it everywhere.

9

u/jnbkadsoy78asdf Jul 05 '21
// helpers/remove-duplicates.js

export default function(array) {
    return [...new Set(array)];
}

Something like that? Looks fancy.

2

u/steezefries Jul 06 '21

Oh cool bro could you go ahead and imagine what that function definition might look like bro?