r/javascript • u/drumnation • Jan 03 '18
Built an SMS bot with Twilio, NodeJS, PhantomJS, Selenium Webdriver and hosted on Heroku. It texts my wife when groceries I like go on sale. Wrote a guide with lots of code examples to jumpstart your own bot project.
https://medium.com/@davidmieloch/building-a-foodkick-sms-sale-notifier-bot-64106c9de9455
u/amgin3 Jan 03 '18
Good effort, but you could do a lot to improve your code. For instance, you don't need to await non-promise functions like console.log or Date(). Also, try moving const declarations outside of your loops when they don't need to be in.
4
u/drumnation Jan 03 '18
Good call. I was a bit overzealous with the awaits while getting it to work. Will go back and clean it up.
9
Jan 03 '18 edited Jan 03 '18
You'd have had an easier time reversing FoodKick's mobile API, as it's exposed in the request traffic when you search.
Paring down via Postman, I can get a successful response from this:
POST https://www.foodkick.com/mobileapi/v/1/search/
Headers:
Content-Type: application/x-www-form-urlencoded
x-fd-extra-response: EXCLUDE_ADDRESS
POST body:
data={"pageType":"SEARCH","searchParams":"blackberries"}Node example:
const fetch = require('node-fetch');
const FormData = require('form-data');
module.exports.search = (terms) => {
let data = { pageType: 'search', searchParams: terms.join(' ') };
let form = new FormData();
form.append('data', JSON.stringify(data));
let headers = form.getHeaders();
headers['x-fd-extra-response'] = 'EXCLUDE_ADDRESS';
return fetch('https://www.foodkick.com/mobileapi/v/1/search/', {
method: 'POST',
body: form,
headers: headers
}).then((result) => result.json()).then((data) => {
'browse.sections.sections'.split('.').forEach((name) => {
data = (data || {})[name];
});
if (!data) {
throw new Error("Malformed response");
}
return data;
});
};
module.exports.isSale = (product) => {
return product.savingString || // Visual flag on bottom
product.dealInfo || // Visual flag at top
(product.availMatPrices.length > 1) || // Means there's a volume price
(product.price > product.wasPrice); // Means there's a markdown
};
module.exports.format = (product) => {
let body = [];
let deal = product.savingString || product.dealInfo;
if (deal) { body.push(deal); }
body.push(product.brandName);
body.push(product.productNameNoBrand);
body.push('$' + product.price + ' (was $' + product.wasPrice + ')');
return body.join('\n');
};Later...
const foodkick = require('./foodkick'); //File described above.
foodkick.search(['blackberries']).then((products) => {
products.forEach((product) => {
if (foodkick.isSale(product)) {
sendSaleInfo(foodkick.format(product), 'blackberries', 'fk');
}
});
});[Note: Attached code is not tested; meant to be a starting point. Postman doesn't manage cookies well so you might need to make a nonsense request somewhere to create one].
[Edit: Freshdirect's API is similar
module.exports.search = (terms) => {
let data = {"pageType":"SEARCH", "searchParams":"blackberries"};
let payload = 'data=' + encodeURIComponent(JSON.stringify(data));
return fetch(
'https://www.freshdirect.com/api/filter?' + payload
).then((result) => result.json()).then((data) => {
'sections.sections'.split('.').forEach((name) => {
data = (data || {})[name];
});
if (!data) {
throw new Error("Malformed response");
}
return data;
});
};The promise result should be very similar. ]
4
u/drumnation Jan 03 '18
Awesome! Will definitely play around with that in case they change the markup and break the scraper.
8
Jan 03 '18
I started something like this - scraping supermarket sites looking for discounts on selected products. It seems they are pretty shrewd in changing their markup to avoid scraping!
2
86
u/basiclaser Jan 03 '18
have you tried talking to your wife
53
7
0
3
u/Silhouette Jan 03 '18
If I tried this, I would start getting auto-replies from my wife saying something like "Great, please pick me up [automatically generated list of some of her favourites] while you're there." :-)
2
1
1
u/ScriptingInJava Jan 03 '18
I was genuinely shocked to find this wasn't /r/programmingcirclejerk just from the title alone.
-3
u/Nrdrsr Jan 03 '18
SMS is icky, but the rest is interesting. I'd probably sub in telegram and try it out. Thanks
0
Jan 03 '18
[deleted]
2
u/drumnation Jan 04 '18
You guys aren't wrong. SMS is all those things. The way the script is organized one could easily replace the way the notification is sent with any other method. In this case, my wife wanted a text message. So a text message is what she got.
Some kind of push notification would certainly be better, but I'm not exactly how to do that without writing a native application. She won't have her browser open either so I'm not sure how HTML5 push notifications could work.
1
Jan 04 '18
Assuming you donât have any need to send rich content - and in most cases, you donât - sms works for a lot of people, and works when theyâre up a mountain with one bar of crappy signal, or on a train going in and out of tunnels, and doesnât require them to have a modern phone or any particular app. On top of that, an sms is persistent and significant. I automatically dismiss notifications because their signal to noise ratio is even lower than email. Few enough people or services have my phone number, that an sms is meaningful.
-13
u/icantthinkofone Jan 03 '18
Of course, for those of us who know how to program, you don't need any of those to do the exact same thing better, faster and smaller.
1
-86
u/jnns Jan 03 '18 edited Jan 03 '18
Why the hell does it text your wife? Why does SHE need to know when groceries YOU like are on sale?
You do realise that this project sounds sexist?
39
u/whyherro19 Jan 03 '18
She probably does the grocery shopping and I'm guessing he built this as a joke. Calm down.
-59
u/jnns Jan 03 '18
Even if she is always the one doing the grocery shopping I would expect the title to be
"it texts my wife when groceries we like are on sale"
or better yet
"wrote an app that notifies me when certain items are on sale".
25
u/PM_ME_UR_CHEST_BOBS Jan 03 '18
I get where you're coming from, but pick your battles. You don't know anything about their living situation or relationship. All you know is that it send his wife a text message.
If you'd actually read the post youd see they talk about getting specials they both like as well
-24
u/jnns Jan 03 '18 edited Jan 03 '18
Their living situation is none of my concern.
What I find annoying is that the intent of his app easily could have been phrased in a neutral way that does not feed the stereotype that a woman's job is to do the housework and go grocery shopping for the working husband.
Edit: forgot important negation đ
6
1
u/PM_ME_UR_CHEST_BOBS Jan 03 '18
Pretty certain no one, but yourself, thought that this post was feeding that stereotype. All it really is saying is "I'm keeping my wife alert of when items we both enjoy are on sale"
It doesn't say "it's my wife's sole responsibility to buy it" or "women belong in the kitchen"
25
u/winzippy Jan 03 '18
/r/TwoXChromosomes is that way ->. Even if it is a smidge sexist or some convoluted "microagression," he just wanted to share his joy of programming.
7
7
u/DOG-ZILLA Jan 03 '18
Maybe she also thought it was a good idea? Maybe his wife likes to do the shopping? Like, actually prefers it that way?
12
9
u/shkico Jan 03 '18
You are completely right, why would she need to do the shopping? Her place is in the kitchen :D
2
2
u/icantthinkofone Jan 03 '18
I'm betting it is far, far more likely his wife does the shopping and cooking. He probably doesn't live in your utopian village.
1
-9
u/whingeypomme Jan 03 '18
This is a perfect example of over-engineering.
OMG I'm going to use this, this, this, this, this, a lot of this, and this to do something so simple, just one thing would suffice... I'm such a cooooool hipster, I use Javascript... Worship me, everyone.
I think only a moron would use the 'code examples' from someone who does something like this, and then posts about it for the reddit world to laugh at them.
3
u/paypaypayme Jan 03 '18
I have to agree with you about over engineering, but there's nothing wrong with using JavaScript... Let me guess, you only write code in FORTRAN AND COBOL?
-4
u/whingeypomme Jan 03 '18
you missed my point. i'm not dissing javascript, i'm laughing at how javascript is used. if someone knew what they were doing, most of that could be replaced with - for example - php.
5
0
Jan 03 '18
[deleted]
2
u/drumnation Jan 03 '18
I started with Chromedriver. When I began trying to host the project on Heroku I had a lot of trouble getting Chromedriver to work. I did a bunch of googling and found a discussion thread talking about trouble with selenium and Chromedriver on Heroku and recommended the PhantomJS buildpack and getting that to work.
165
u/dottybotty Jan 03 '18
Lol builds an entire service to text his wife to get him something when he could just get it himself đ