r/Python • • Apr 05 '19

Using python to map the traffic that came to my server after my post hit the front page.

Post image
1.9k Upvotes

74 comments sorted by

88

u/Gedanke Apr 05 '19 edited Apr 05 '19

I wrote a python script that analyses my NGINX access.log and creates this map using d3.js .

This is the post in question if anyone is wondering.

You can find the code @ Github

5

u/kingo86 Apr 06 '19

Is that using geoip from your access logs or are you collecting GPS coordinates?

206

u/mostlowkey1 Apr 05 '19

I’ve been following r/python for a long time just because I’ve been really interested in learning it and I knew seeing various posts would inspire me. There was something about seeing this that made me realize I do think data visualization is super cool and I think you just gave me the final push to get started!

92

u/Gedanke Apr 05 '19

Aww! This is more than I could've hoped to achieve with this post! I am glad. Let me know if you have any questions.

17

u/mostlowkey1 Apr 05 '19

It’s just so simple yet cool! Thanks! I’ll have to come up with the questions first 😅

6

u/Thymester Apr 06 '19

To be fair this is a great post. I like the visuals of it even. You kinda also got me thinking maybe I should give up on Web Development and go to Data Visualization. (And this is in all seriousness not just because the guy above me said something similar)

8

u/[deleted] Apr 06 '19

2

u/mostlowkey1 Apr 06 '19

I know this is a wrong way to think but I’m almost intimidated how big this community is. Makes me feel like I’m almost too late to hop on. I know I should think this way but I’m a little older now and big learning curves are kinda intimidating

2

u/[deleted] Apr 06 '19

Most of the gems act like kickstarters. Bigger the community, better the feedback gets. Your work will of course get appreciated if it suits the subreddit's needs.

1

u/mostlowkey1 Apr 06 '19

Okay cool. Thank you :)

2

u/[deleted] Apr 06 '19

After I've read your comment I thought common, it can't be that big. But damn dude what the hell

2

u/MuhdnurShuaib Apr 06 '19

I'm right behind you, behind you bcs I'm currently learning Django and don't want to get into something else yet.... Want to have a good grasp on it 1st

0

u/[deleted] Apr 06 '19

[deleted]

2

u/mostlowkey1 Apr 06 '19

Thank you!

29

u/MotherfuckerTinyRick Apr 05 '19

I can see my house from here

19

u/Gedanke Apr 05 '19

That's amazing. The data is rasterised to ~160x160 km squares, so your actual location should be hidden.

25

u/ChinaFlavoredWater Apr 06 '19

Nah I can see it if I zoom in and ‘enhance’ the image.

6

u/RedRidingHuszar Apr 06 '19

CSI style?

4

u/Kossak Apr 06 '19

No, gandam style

2

u/peacounter Apr 06 '19

You‘ve got a big ..... house. And even better eyes.

5

u/MotherfuckerTinyRick Apr 06 '19

It's a big red dot in the middle of nowhere

3

u/What_Is_X Apr 06 '19

Alice Springs?

2

u/WeAmGroot Apr 06 '19

Hermannsburg

20

u/[deleted] Apr 05 '19

Did you have trouble mapping hits from China?

6

u/RedRidingHuszar Apr 06 '19

Could it be some sort of nationwide VPN?

9

u/[deleted] Apr 06 '19

No it's probably that the GeoIP database that the OP used maps all Chinese IPs to the same location, because it could be hard to retrieve more precise info.

The locations in such tables are often quite crude, based on data that's aggregated from various sources and IP lookups.

If you have an IP you know to which country it belongs because you can simple check which provider registered it in which country (for example at bgp.he.net, but this is publicly available info). If the provider happens to be active in one province/state only you can narrow it down further.

For a more precise lookup you can trace an IP, which will give you more info about the area. For example, when I trace your IP in Canada I may get a path like Home -> regional hub -> regional hub -> Amsterdam -> Ashburn/NYC -> Toronto -> regional hub -> ... -> your home. If from elsewhere I know the last regional hub is in Montréal, then I'm quite sure you're living in the proximity of Montréal and that's the closest we can get.

Such information is worth gold, as it can be used for targeted advertising without even knowing who the person is you're targeting.

It's very plausible that for China no such info is collected by the GeoIP database. Maybe it's hard to pinpoint IPs to locations because the IP space is changing a lot every day, or maybe China is just not interesting to target for the databases.

2

u/greenkiweez Apr 06 '19

Big brother is always watching. I wonder if using a site like Reddit is legal there.

4

u/port53 relative noob Apr 06 '19

It's only legal to buy, not use, Reddit in China.

11

u/[deleted] Apr 06 '19

South Korea loved it! North Korea not so much.

12

u/[deleted] Apr 05 '19

What's that one point in central greenland? There should be nothing but ice afaik.

3

u/[deleted] Apr 06 '19

Probably where whatever he used for the geoIP data defaults to if it doesn't know the precise location in a country (i.e. the centre of Greenland). You can see a similar thing in China where there's a large amount of traffic from the centre when there's no major population there.

5

u/GuitarPerson159 Apr 06 '19

snowman

3

u/ohaiya Apr 06 '19

that's abominable.

6

u/jhdeval Apr 05 '19

I think it would interesting to see time of day as well. Not sure if it would work but you may see a wave as it traverses the timezones.

4

u/piquat Apr 05 '19

Yes. An animation with each dot popping out as the clock runs on screen. Don't know how hard that would be with python, I'm just a newb.

7

u/jhdeval Apr 05 '19

There's only 24 hours in a day may e a different color for each hour rather then a rolling view.

1

u/piquat Apr 05 '19

Ya, that would probably work and be much easier to implement. I'd bet you'd still get a good sense of it rolling across the planet.

5

u/zandiz Apr 06 '19

I want to know who that one nerd in Libya is

4

u/ihitcows Apr 06 '19

That one dot in China really stands out.

3

u/liesliesfromtinyeyes Apr 06 '19

You have a point in what could be Punta Arenas, Chile. I wonder if that’s an American down for deployment to Antarctica with the US Antarctic Program...

5

u/[deleted] Apr 06 '19

Australia uses reddit? I thought they were too busy: surfing, bagging kangaroos, downing exotic(local for them) beers, working on that tan/bod, avoiding insert creepy insect, avoiding insert typical plague, avoiding insert animal that eats you, etc etc....

7

u/D4rk_unicorn Apr 06 '19

You tryna get pulped m8

2

u/[deleted] Apr 06 '19

...avoiding a pulping from a m8, going to pulp a m8...

2

u/ElGallinero Apr 05 '19

What was the post?

2

u/Unclerojelio Apr 06 '19

How very meta of you.

2

u/lord_tommy Apr 06 '19

These little tiny dots make me curious about what else they look at. What do they enjoy browsing all the way North in Canada? That dot in the middle of South America all by itself, what are they looking through reddit for? It just makes me so curious what life is like in those seemingly quieter traffic parts of the world.

2

u/hamstap85 Apr 06 '19 edited Apr 07 '19

Interesting. This looks just like a heat map of internet usage.

2

u/avinashbasetty Apr 06 '19

This is interesting..what are the APIs you have used to achieve this?

1

u/Long__Dog Apr 05 '19

Perhaps you could look at excluding areas in the sea i.e. The English Channel, between France and England which looks like one of the 'hot spots'

1

u/[deleted] Apr 06 '19

Reddit isn't banned in China? I didn't know that

1

u/nikhil_shady Apr 06 '19

ahaha I'm the only one with a dot from my city.

1

u/shardul08 Apr 06 '19

Where will I get the access.log file?

1

u/lenticularis_B Apr 06 '19

I did not know you could extract location from an ip-addres. You might use this information to figure out the optimal time for posting.

3

u/grumbelbart2 Apr 06 '19

It's often called "GeoIP". IPs are usually allocated to certain networks, and the location of those networks is often known or approximated from whois information, ping triangulation and other techniques.

It's not always very precise, for example, when dynamic IPs are re-assigned in a large networks.

It's the same technique used by porn sites to show "women near you" weather sites to show the weather at your location.

2

u/lenticularis_B Apr 06 '19

A that makes sense! Takes away some magic around ip-localization, thanks.

1

u/MetalAvenger Apr 06 '19

Saved post for when I get beyond python fundamentals... :)

-1

u/mardymole Apr 06 '19

I wouldn’t use this as inspiration. The code is terrible

1

u/barnabas09 Apr 06 '19

europe and USA huh?

1

u/[deleted] Apr 06 '19

Most of Africa is in the dark

1

u/Petrarch1603 Apr 06 '19

I wonder if that big red spot in Kansas is this place

1

u/Petrarch1603 Apr 06 '19

you should crosspost this to /r/MapPorn!

1

u/[deleted] Apr 06 '19

Kenya

1

u/quickhakker Apr 06 '19

Could you use that to figure out when the post hit the front page?

1

u/cronched Apr 09 '19

Is the one single dot in the ocean next to madagascar alright? lol

1

u/jsalsman Apr 05 '19

3

u/a23y1 Apr 06 '19

I don't know if you've checked the populations of India and China recently, but there's a bit of a discrepancy between their populations and what's shown in the image.

1

u/jsalsman Apr 06 '19

Hover for the title text.

3

u/a23y1 Apr 06 '19

India is only second to the US with regards to the number of English speakers. English isn't their native language, but neither is it in most of Europe.

Source: https://en.wikipedia.org/wiki/List_of_countries_by_English-speaking_population

3

u/[deleted] Apr 06 '19

Sure but what is important is the percent English speakers, in which case all of Europe is higher than India (except Russia)

0

u/Lowbacca1977 Apr 06 '19

Only if you normalized your data to be what fraction of the population visited your site for this.

0

u/farooq_fox Apr 06 '19

You are being watched by the chinese government.

0

u/mrwill1 Apr 06 '19

Someone in China is using VPN.

-3

u/atands Apr 06 '19

I don’t mean this in a rude way — I think what you did is cool, but what’s the reason that you didn’t just use an analytics tag/service for this?