r/web_infrastructure • • Mar 09 '17

The System Design Primer - Learn how to design large scale systems

Thumbnail
github.com
10 Upvotes

r/web_infrastructure • • Oct 29 '19

Lights Out: Climate Change Risk to Internet Infrastructure [PDF]

Thumbnail pages.cs.wisc.edu
6 Upvotes

r/web_infrastructure • • Jun 25 '10

Digg Technology Blog (Please don't downvote, it is a legitimate resource)

Thumbnail
about.digg.com
6 Upvotes

r/web_infrastructure • • May 16 '16

NOW TV: Linear Streaming and the Unpredictable Scalability Challenge

Thumbnail
developers.sky.com
5 Upvotes

r/web_infrastructure • • Mar 26 '12

7 years of Youtube scalability lessons in 30 minutes

Thumbnail
highscalability.com
5 Upvotes

r/web_infrastructure • • Jun 09 '08

Scaling Fast: Result of being on the front page of Yahoo.com

Thumbnail lukasbiewald.com
5 Upvotes

r/web_infrastructure • • Aug 11 '17

Question about scalability article

4 Upvotes

Regarding the following article: http://highscalability.com/blog/2016/1/11/a-beginners-guide-to-scaling-to-11-million-users-on-amazons.html Does anyone know how #users are counted - is it concurrent, total, per hour, or other? Thanks!

ps - I have posted to r/aws and r/programming and gotten two different answers so far - 1. Monthly 2. Concurrent (makes a big difference)


r/web_infrastructure • • Jul 19 '17

Looking for a file management and storage system like Windows Explorer.

4 Upvotes

My team and I have a lot of content to work with regarding e-learning files. We need a system that can handle large storage sizes, especially with various video files. We like how Windows Explorer on our company network handles this, but we need to find a separate place to store these files in order to backup.


r/web_infrastructure • • Nov 19 '15

Goofys 0.0.3 released - a faster s3fs written in Go

Thumbnail
github.com
4 Upvotes

r/web_infrastructure • • Jun 26 '08

A Look Inside Wikipedia's Infrastructure - Data Center Knowledge

Thumbnail
datacenterknowledge.com
4 Upvotes

r/web_infrastructure • • Jul 23 '24

Storage + Postgres + Vector store?

3 Upvotes

I need to store 10gb+ of PDFs, along with their plain text and metadata, as well as some 1.5M vectors for a semantic retrieval system. The DB will almost only handle reads.

At first I went with Supabase, as they offer all that in a fully manages fashion, but given the size of the DB, I can't go with th free plan, and 25$/m seems overkill, especially since I will not be using auth or realtime functionalities, which are where Supabase shines.

So I took the cheap, dirty path with a $5/m Contabo VM where I'm self hosting a postgresql + pgvector. Problem is I'm not sure how reliable this infrastructure is, and the latency is not great since I'm in South America, and the closest Contabo servers are in NA.

Now, I don't need a super fast service, but I was wondering if there are better (affordable) options for my requirements, which basically boil down to low CPU, low memory, but (somewhat) bigger storage and reliability.

Thanks


r/web_infrastructure • • Jul 23 '24

Progressive Web Apps vs. Native Apps: Which Works Best for Your Business?

Thumbnail
quickwayinfosystems.com
3 Upvotes

r/web_infrastructure • • Aug 13 '21

Should you use AWS Route 53 for both your domains and subdomains or use it only for one of them?

3 Upvotes

We have a domain on GoDaddy and planning on routing the traffic to it through Route 53 and later will be creating subdomains using Route 53 too. So I wanted to know what are the pros and cons and also the security concerns for both scenarios of using GoDaddy for only the domain and Route 53 for subdomains and vice versa.


r/web_infrastructure • • May 27 '20

Question about deploying python script on the cloud

3 Upvotes

Hi,

So, I'm developing a python script that will eventually be hosted on a website. Users can enter different parameters to run script that is essentially a bot running 24/7. I've been looking for cloud solutions that are cost efficient (AWS, Google, Python anywhere) , but I've never used such service and don't know how to calculate the cost.

Essentially, every user that sign ups will have their own bot running by its own, so what would be the best way to deploy something like this and calculate cost?


r/web_infrastructure • • May 05 '18

Question : How to store and process streaming data in real time ?

3 Upvotes

I have an app which sends a lot of real time data to my servers. Each packet is about 5 floats, but it is transmitted at the rate of 10-50 Hertz.

I want to store this data for a set period of time, like 30 minutes. I might need longer term storage in the future.

I want to process this stream of data in real time, and making sure that this has low latency is absolutely crucial.

How can I build a scalable architecture for dealing with these requirements ?

I was think about using Cassandra to store the time series data. Kafka to introduce redundancy, but I am not sure about it. Does having a Kafka layer introduce latency ?

One way I was thinking about doing this is to, do the necessary processing on the data, on the server that actually accepts the packets from the user. It is not complete heavy, so I think it is doable. And then from that server, also forward those packets to Cassandra.

Does this make sense ?

How would things change if I wanted to send more data. Rather than it been 5 floats, I send a whole image. Streaming images, basically a video. Would it still make sense to store the data in Cassandra ?


r/web_infrastructure • • Sep 17 '17

An AWS Lambda Function to clean ecr automatically

3 Upvotes

Hi, All!

I've created lambda pack for cleaning up old Docker images from ECR/ECS I will be glad to get some feedback 👌

https://github.com/puemos/aws-lambda-ecr-cleaner


r/web_infrastructure • • Aug 22 '17

Question: Best practice to realize an autocomplete in 2017?

3 Upvotes

How to realize an autocomplete with hundreds of options? Should it be an API Endpoint with caching & invalidation? A Redis cache? Or a local storage with some form of invalidation? What else?


r/web_infrastructure • • Aug 08 '17

Looking for a web server tunnel to CLI

3 Upvotes

Hi everyone. I am new to the sub, but hope my topic is not "off-topic" here :).

I am looking for a web interface tunnel to any CLI application (just take telnet as example). I have stumbled across one a long while ago, before websockets (which would be great for this) were a thing and all. Of course I can't remember the name...

What it should be capable of doing is, start a CLI application and reroute the TTY to my browser, and in the first step I want to use it like my shell on the server. I'd like it if the proxy can be modified by me (I know a bit of css, javascript and various backend languages) so I can style and put alerts on the TTY output.

Anyone got some hints? Thanks!


r/web_infrastructure • • May 08 '17

Self-service provisioning can accelerate testing and programming efforts while reducing IT management and development costs. But only if you pay attention during its adoption.

Thumbnail
insights.hpe.com
3 Upvotes

r/web_infrastructure • • Jan 07 '17

Sfs: OpenStack Swift API and Haystack Distributed Object Store written in Vert.x

Thumbnail
github.com
3 Upvotes

r/web_infrastructure • • Apr 25 '16

Dashboard on top of REST data feeds?

3 Upvotes

I'm not sure if this is the right subreddit or not but here it goes.

I'm working on a small IoT project and I'm in need of a dashboard that allows me to feed it data from a RESTful API. What I have in mind is something fairly out-of-the-box that gives components, such as tabular log views, charts, etc. all that can be fed from an API.

Does anyone know of any project that allows for this that is easy to host and work with? If this is the wrong subreddit let me know and I can move it to a more appropriate one.

-S


r/web_infrastructure • • Sep 03 '15

Why you should stop using the us-standard Region in S3. Right Now.

Thumbnail instrumentalapp.com
3 Upvotes

r/web_infrastructure • • Mar 03 '15

Looking for advice on latest whitepapers for best practices re: security

3 Upvotes

Hey /r/web_infastructure! I was looking for any updates resources regarding the latest in security best practices with web development these days, specifically aws. I've read all of the whitepapers that they have available, but they're all circa 2013 off of their blog.

I meant to add that i've read through most of https://gist.github.com/chanj/6c48c059ad4b72a60bf3

Any other specific tools, resources, or case studies you would suggest looking into?

Thanks!


r/web_infrastructure • • Nov 22 '13

How we've implemented simple caching with Redis and Python, replacing memcached and optimizing the query layer of our music platform!

Thumbnail
blog.seevl.fm
3 Upvotes

r/web_infrastructure • • Jun 21 '08

Cutting the Cat 5 Cord? Not So Fast!

Thumbnail blogs.channelinsider.com
3 Upvotes