r/javascript • • Jan 03 '18

Built an SMS bot with Twilio, NodeJS, PhantomJS, Selenium Webdriver and hosted on Heroku. It texts my wife when groceries I like go on sale. Wrote a guide with lots of code examples to jumpstart your own bot project.

https://medium.com/@davidmieloch/building-a-foodkick-sms-sale-notifier-bot-64106c9de945
379 Upvotes

73 comments sorted by

165

u/dottybotty Jan 03 '18

Lol builds an entire service to text his wife to get him something when he could just get it himself 😂

10

u/qualei Jan 03 '18

Does the cost of running the service cost more than the discounts?

12

u/drumnation Jan 03 '18

Everything is free except the texts. I have to admit that I built this fully expecting to reuse this code for something else so I know it's a little bit overkill.

2

u/qualei Jan 03 '18

Good work!

1

u/Devboe Jan 03 '18

Couldn’t you send an email to the phone number for free?

1

u/Disowned Jan 03 '18

Each mobile provider has a different email address you would have to send it to. You would have to know what provider the phone number is connected to in advance.

2

u/Devboe Jan 03 '18

If he’s sending it to his wife only, then he would know the mobile provider. I can see how this wouldn’t be ideal in all circumstances, but for OP specifically, it seems like it’s viable if he wanted no cost involved.

1

u/Disowned Jan 03 '18

I'm approaching it from an agnostic point of view, where we don't assume anything about the end user. It's assumed he would know he's wife's phone provider. However, since he has also said he wanted to provide his work for reuse for other people you can't make the assumption that this application is going to be used by a platform that just happens to know what the end user mobile provider is unless they specifically asked for it. Even then, using the email method could fail because you have a finite amount you could send to that phone number since you wouldn't be using an authorized gateway (which most of these services are.) Not to mention, you would also have to have that email correct on the application end. Abstracting this out to a service dedicated to doing this takes all of that busy work out of the equation.

1

u/DemiPixel Jan 07 '18

Half a cent to check the carrier and all texts are free VS 3/4 cent per message, hmmmmm

https://www.twilio.com/lookup

Or just use telegram as mentioned above.

1

u/Disowned Jan 07 '18

It's not that big of a difference honestly. .005 vs .0075 Just pay the extra .0025 and save yourself some cycles.

1

u/DemiPixel Jan 07 '18

It's $0.005 for a phone number which you could send 100 messages to. That would cost you $0.75 in messages.

→ More replies (0)

1

u/Askee123 Jan 03 '18

I think it's only 0.004 cents per text.

10

u/cag8f Jan 03 '18

It might not sound helpful, but it actually lays the groundwork for something that could be helpful to frugal people. One can adapt this project to email you every time a weekly sale at one of your local grocery stores brings the price of chicken breast below $2.99/lb. Or whenever toilet paper falls below $X per roll. Any item you want and any price threshold you want. Then you go stock up on those items, saving yourself money in the long run.

OP, a few years ago I began to build something similar, for grocery stores in my region. I had ideas to make it an app that scanned grocery stores nationwide (at least, those that posted their weekly specials online), and gave users the option to monitor any item at any store(s). But the scraping process would have to be tailor made for each individual grocery store. If I recall, even branches within a chain of stores (e.g. Giant, Safeway, etc) had slightly different website layouts in terms of DOM structure. In that case, it would be very difficult (i.e. require a lot of man hours) to stay abreast of changes to any specific website, and fix your scraping algorithm. I guess you could automate the process of monitoring websites for DOM changes. But you would still have then in-turn manually fix the scraping algorithm for that site.

25

u/[deleted] Jan 03 '18

You’re missing the point of the comment.

He built something to tell his wife to go get it. It’s not messaging himself or making a list for him to refer to.

It’s coldly assigning errands. To his wife.

It’s hilariously tone deaf.

6

u/drumnation Jan 04 '18 edited Jan 04 '18

I should have titled this post with Food "We" Like instead of Food "I" like. I can see how the title might appear chauvinist without that tweak.

Right now my wife works for the Rachael Ray TV show, but for several years she worked for the magazine and wrote about saving money at the grocery store. She asked me to code this for her to help optimize our grocery shopping.

One Friday night, we were drinking really good margaritas at home and she realized that she bought these mixers all at half price… “Damn, I would buy these every single time they go on sale.”

The programmer in me just couldn’t resist trying to automate the “checking if the margarita mix is on sale.”

12

u/butwhysheasked Jan 03 '18

Maybe he and his wife divide responsibilities, as opposed to randomly assigning roles to themselves as opportunities arise. There are plenty of other relationship and homeowner responsibilities, whose ownership is not mentioned in this article. His role in this case is “saving money their family can put to better use”, which is not to be looked down upon.

11

u/tehstone Jan 03 '18

You're making a lot of assumptions there.

5

u/robvert Jan 03 '18

By wife he means robot that he will build in part 2 of the series

3

u/username223 Jan 04 '18

"Part 2: How I glued a sex doll to a Segway" (it has to be able to roll to the store and buy the discount toilet paper, after all...)

5

u/DivineVodka Jan 03 '18

Here you are getting offended for someone's wife. Like I hate saying white knight and all, but do you understand the dynamics of their relationship? Holy moley stop projecting, and comment on the CODE, you know the thing this sub is about.... Jesus lmao.

4

u/loratliff Jan 03 '18

Well, given that FoodKick is a grocery delivery service in NYC, no one is running an errand—they're getting a text, loading some items in a virtual cart, and checking out.

0

u/NooJoisey Jan 03 '18

Ahh.. the entire world lives in 300 square miles.

4

u/loratliff Jan 03 '18

If you actually read his article...

-1

u/whingeypomme Jan 03 '18

It lays the groundwork for how not to code.

34

u/saitilkE Jan 03 '18

Relationship goals

1

u/Bilddalton Jan 03 '18

It's funny lol

1

u/icantthinkofone Jan 03 '18

Except in the more likely case that she does the grocery shopping and cooking.

5

u/amgin3 Jan 03 '18

Good effort, but you could do a lot to improve your code. For instance, you don't need to await non-promise functions like console.log or Date(). Also, try moving const declarations outside of your loops when they don't need to be in.

4

u/drumnation Jan 03 '18

Good call. I was a bit overzealous with the awaits while getting it to work. Will go back and clean it up.

9

u/[deleted] Jan 03 '18 edited Jan 03 '18

You'd have had an easier time reversing FoodKick's mobile API, as it's exposed in the request traffic when you search.

Paring down via Postman, I can get a successful response from this:

POST https://www.foodkick.com/mobileapi/v/1/search/

Headers:
  Content-Type: application/x-www-form-urlencoded
  x-fd-extra-response: EXCLUDE_ADDRESS

POST body:
  data={"pageType":"SEARCH","searchParams":"blackberries"}

Node example:

const fetch = require('node-fetch');
const FormData = require('form-data');

module.exports.search = (terms) => {
  let data = { pageType: 'search', searchParams: terms.join(' ') };
  let form = new FormData();
  form.append('data', JSON.stringify(data));
  let headers = form.getHeaders();
  headers['x-fd-extra-response'] = 'EXCLUDE_ADDRESS';
  return fetch('https://www.foodkick.com/mobileapi/v/1/search/', {
      method: 'POST',
      body: form,
      headers: headers
    }).then((result) => result.json()).then((data) => {
      'browse.sections.sections'.split('.').forEach((name) => {
        data = (data || {})[name];
      });
      if (!data) {
        throw new Error("Malformed response");
      }
      return data;
    });
};
module.exports.isSale = (product) => {
    return product.savingString || // Visual flag on bottom
           product.dealInfo || // Visual flag at top
           (product.availMatPrices.length > 1) || // Means there's a volume price
           (product.price > product.wasPrice); // Means there's a markdown
};
module.exports.format = (product) => {
    let body = [];
    let deal = product.savingString || product.dealInfo;
    if (deal) { body.push(deal); }
    body.push(product.brandName);
    body.push(product.productNameNoBrand);
    body.push('$' + product.price + ' (was $' + product.wasPrice + ')');
    return body.join('\n');
};

Later...

const foodkick = require('./foodkick'); //File described above.

foodkick.search(['blackberries']).then((products) => {
    products.forEach((product) => {
        if (foodkick.isSale(product)) {
            sendSaleInfo(foodkick.format(product), 'blackberries', 'fk');
        }
    });
});

[Note: Attached code is not tested; meant to be a starting point. Postman doesn't manage cookies well so you might need to make a nonsense request somewhere to create one].

[Edit: Freshdirect's API is similar

module.exports.search = (terms) => {
  let data = {"pageType":"SEARCH", "searchParams":"blackberries"};
  let payload = 'data=' + encodeURIComponent(JSON.stringify(data));
  return fetch(
    'https://www.freshdirect.com/api/filter?' + payload
  ).then((result) => result.json()).then((data) => {
    'sections.sections'.split('.').forEach((name) => {
        data = (data || {})[name];
    });
    if (!data) {
      throw new Error("Malformed response");
    }
    return data;
  });
};

The promise result should be very similar. ]

4

u/drumnation Jan 03 '18

Awesome! Will definitely play around with that in case they change the markup and break the scraper.

8

u/[deleted] Jan 03 '18

I started something like this - scraping supermarket sites looking for discounts on selected products. It seems they are pretty shrewd in changing their markup to avoid scraping!

2

u/drumnation Jan 03 '18

We'll see how long it lasts before I need to change the scraping code :)

86

u/basiclaser Jan 03 '18

have you tried talking to your wife

7

u/saitilkE Jan 03 '18

Where's the fun in that though?

0

u/[deleted] Jan 03 '18

women will make you build javascript bots in order to avoid hearing them

3

u/Silhouette Jan 03 '18

If I tried this, I would start getting auto-replies from my wife saying something like "Great, please pick me up [automatically generated list of some of her favourites] while you're there." :-)

2

u/dangoodspeed Jan 03 '18

Your wife sounds like a lucky lady!

1

u/digdic Jan 04 '18

wow this sub is just as bad as /r/node when it comes to shitlords huh

1

u/ScriptingInJava Jan 03 '18

I was genuinely shocked to find this wasn't /r/programmingcirclejerk just from the title alone.

-3

u/Nrdrsr Jan 03 '18

SMS is icky, but the rest is interesting. I'd probably sub in telegram and try it out. Thanks

0

u/[deleted] Jan 03 '18

[deleted]

2

u/drumnation Jan 04 '18

You guys aren't wrong. SMS is all those things. The way the script is organized one could easily replace the way the notification is sent with any other method. In this case, my wife wanted a text message. So a text message is what she got.

Some kind of push notification would certainly be better, but I'm not exactly how to do that without writing a native application. She won't have her browser open either so I'm not sure how HTML5 push notifications could work.

1

u/[deleted] Jan 04 '18

Assuming you don’t have any need to send rich content - and in most cases, you don’t - sms works for a lot of people, and works when they’re up a mountain with one bar of crappy signal, or on a train going in and out of tunnels, and doesn’t require them to have a modern phone or any particular app. On top of that, an sms is persistent and significant. I automatically dismiss notifications because their signal to noise ratio is even lower than email. Few enough people or services have my phone number, that an sms is meaningful.

-13

u/icantthinkofone Jan 03 '18

Of course, for those of us who know how to program, you don't need any of those to do the exact same thing better, faster and smaller.

1

u/whingeypomme Jan 03 '18

but i bet you don't use javascript.

1

u/icantthinkofone Jan 03 '18

Haven't a clue why you would think that.

-86

u/jnns Jan 03 '18 edited Jan 03 '18

Why the hell does it text your wife? Why does SHE need to know when groceries YOU like are on sale?

You do realise that this project sounds sexist?

39

u/whyherro19 Jan 03 '18

She probably does the grocery shopping and I'm guessing he built this as a joke. Calm down.

-59

u/jnns Jan 03 '18

Even if she is always the one doing the grocery shopping I would expect the title to be

"it texts my wife when groceries we like are on sale"

or better yet

"wrote an app that notifies me when certain items are on sale".

25

u/PM_ME_UR_CHEST_BOBS Jan 03 '18

I get where you're coming from, but pick your battles. You don't know anything about their living situation or relationship. All you know is that it send his wife a text message.

If you'd actually read the post youd see they talk about getting specials they both like as well

-24

u/jnns Jan 03 '18 edited Jan 03 '18

Their living situation is none of my concern.

What I find annoying is that the intent of his app easily could have been phrased in a neutral way that does not feed the stereotype that a woman's job is to do the housework and go grocery shopping for the working husband.

Edit: forgot important negation 😔

6

u/[deleted] Jan 03 '18

Yeah and that's why it's funny, stereotypes are great.

1

u/PM_ME_UR_CHEST_BOBS Jan 03 '18

Pretty certain no one, but yourself, thought that this post was feeding that stereotype. All it really is saying is "I'm keeping my wife alert of when items we both enjoy are on sale"

It doesn't say "it's my wife's sole responsibility to buy it" or "women belong in the kitchen"

25

u/winzippy Jan 03 '18

/r/TwoXChromosomes is that way ->. Even if it is a smidge sexist or some convoluted "microagression," he just wanted to share his joy of programming.

7

u/Yerfacemate Jan 03 '18

It does that because that's what he wants it to do.

7

u/DOG-ZILLA Jan 03 '18

Maybe she also thought it was a good idea? Maybe his wife likes to do the shopping? Like, actually prefers it that way?

12

u/NahroT Jan 03 '18

T R I G G E R E D

9

u/shkico Jan 03 '18

You are completely right, why would she need to do the shopping? Her place is in the kitchen :D

2

u/moreanswers Jan 03 '18

You don't know, maybe he has a 10 foot dong?

2

u/icantthinkofone Jan 03 '18

I'm betting it is far, far more likely his wife does the shopping and cooking. He probably doesn't live in your utopian village.

1

u/Nrdrsr Jan 03 '18

Suffer. I live for this

-9

u/whingeypomme Jan 03 '18

This is a perfect example of over-engineering.

OMG I'm going to use this, this, this, this, this, a lot of this, and this to do something so simple, just one thing would suffice... I'm such a cooooool hipster, I use Javascript... Worship me, everyone.

I think only a moron would use the 'code examples' from someone who does something like this, and then posts about it for the reddit world to laugh at them.

3

u/paypaypayme Jan 03 '18

I have to agree with you about over engineering, but there's nothing wrong with using JavaScript... Let me guess, you only write code in FORTRAN AND COBOL?

-4

u/whingeypomme Jan 03 '18

you missed my point. i'm not dissing javascript, i'm laughing at how javascript is used. if someone knew what they were doing, most of that could be replaced with - for example - php.

5

u/paypaypayme Jan 03 '18

If you said bash I would have taken you seriously

0

u/[deleted] Jan 03 '18

[deleted]

2

u/drumnation Jan 03 '18

I started with Chromedriver. When I began trying to host the project on Heroku I had a lot of trouble getting Chromedriver to work. I did a bunch of googling and found a discussion thread talking about trouble with selenium and Chromedriver on Heroku and recommended the PhantomJS buildpack and getting that to work.