I recently reached out to Digital Ocean Support because my Spaces Bandwidth charges were higher than I expected. For a while i'd simply sent any user wishing to download a file from my website to a short lived signed URL at Digital Ocean and not thought much more of it. The costs slowly increased alongside my traffic and I just accepted that I was using the amount of bandwidth Digital Ocean told me I was in my monthly bill.
However, when investigating another unrelated issue with my site I came across something strange. I was logging every single download (or at least every time someone clicked the link that redirected them to the Spaces url) and the sum of the size of all the files that had been downloaded was significantly less than the amount of bandwidth I was seeing in my bill and the Digital Ocean console.
Of course my logging wasn't going to be accurate, the bandwidth to download a file may be more than the size of the file and users may restart or retry a download (the signed URLs expired after 5 minutes).
I reached out to support and they told me they are unable to share access logs and can only give me a total bandwidth usage for the past 7 days at a time.
There was some initial confusion as when you ask for support for your Spaces Bandwidth Usage they will provide you with a figure based solely on Spaces, without the use of the CDN, but when you look in the console or your bills these two figures are combined and described solely as "Spaces Bandwidth Overage".
But once I got both figures and combined them, I became very worried.
For the 7 days in question I was told the bandwidth usage was 5,566 GB for spaces and 7,849 GB for CDN for a total of 13,415 GB.
But my internal logging only showed 5,265 GB usage, i'm being charged 2.5x more than my own logs suggest I should!
I began to worry that somebody had access to my Spaces buckets directly, but since it's all just a service I don't have access to any access logs or anything like that to try and debug this.
DO support wasn't particularly helpful either, they couldn't provide any access logs or times, only daily bandwidth usage for the past 7 days, not particularly helpful...
I spent some time trying to understand why I was being charged for both Spaces and CDN when I was sending 100% of my users to the CDN. Surely I should have no charges for Spaces at all?
I was also becoming infuriated with the fact that every page related to Spaces and CDN on the Digital Ocean website claimed "the free, built-in Spaces CDN minimizes page load times, improves performance, and reduces bandwidth and infrastructure costs."
Was this just a blatant lie? It definitely didn't seem to be true for me.
Confusingly DO support kept telling me that the CDN was FREE. How could this be true when you're giving me my the costs of using the CDN and they're higher than my costs for using Spaces!
DO support informed me that when using the CDN I would only be charged for the request from Spaces to the CDN the first time, after that the CDN would be caching the files and I wouldn't be charged anything.
If the CDN is free then the bandwidth usage should be less than the total sum of all the files that have been downloaded, but this obviously isn't the case.
After multiple days trying to understand this and setting up experiments with different buckets I finally figured it out.
On this How to Manage the Spaces CDN Cache help page it says the following:"Each request with a unique URL, including the URL’s query string, is treated as a unique asset and has its own cache"When I first skimmed past this bit I assumed it was just a little tip for cache busting. If you update a file and don't want to risk the user accessing a cached version, simply add a query string with a timestamp to bypass the cache, i'd been doing something similar with CSS and JS files for more than decade.
But the it dawned on me, the URLs i'm generating are all signed, so they all look like this:
https://bucketname.nyc3.cdn.digitaloceanspaces.com/myfile.zip?response-content-disposition=attachment%3B%20filename%3D%22myfile.zip%22&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=2QHBRGVPPKQPDLH7GUUJ%2F20220309%2Fnyc3%2Fs3%2Faws4_request&X-Amz-Date=20220309T021234Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Signature=2d765a691cdc9178146c49cd57a47287e2ebe685c1651c7066754cc650e44f1f
The URL has a query string!
But surely signed URLs don't invalidate the CDN cache, that would be absurd and make the CDN completely pointless.
This was probably my biggest mistake. It would make it pointless for my use case, but I forgot that other people had other use cases.
I've always used Spaces (and AWS S3) to offer files for download protected by signed URLs. But of course, some people would use it for hosting assets such as images and CSS that don't require this sort of protection.
I've never considered allowing direct access to my Spaces, if that were to happen it would be a huge security flaw. It's such a no-no that I forgot that some people would want to work that way, alot of people in fact, and all the Spaces documentation was geared towards those people and not me.
Using signed URLs isn't mentioned in the main pages advertising Spaces and even in the documentation it's not mentioned at all other than in the specific page explaining how to use Spaces with the AWS S3 SDK. It's easy to miss.
So what was happening was that every download was being treated like a unique file. So when I sent a user to download a file using the CDN, the CDN cache would be missed so the CDN would request the file from Spaces, incurring the bandwidth cost for the file and I would also be charged for the bandwidth cost of the file being sent to the end user.
I was being charged double after all!
I didn't need to stop using signed URLs, I just had to stop using the CDN. I'm not even sure what benefit the CDN would be giving if the cache was being missed 100% of the time, possibly none?
Once I did this my costs dropped from $1291.82 in February to $212.24 in March, that's a drop of 85%!
I suggested to Digital Ocean that they make this clearer in their documentation but so far they haven't. The documentation for Signed URLs gives explicit instructions on how to use the CDN with signed URLs with no mention of the increased costs.
I'm not sure how to test if the CDN is making any difference with a signed URL. I downloaded both at the same time using a CDN from an office space in Mexico and they both downloaded at 2.5 MB/s. But this test was obviously not perfect.
Either way, if you're using CDN and signed URLs I suggest you take a look at how much you could save by switching away from the CDN.
I would love it if the Digital Ocean CDN was updated to ignore query strings related to Signed URLs, so that we could benefit from both the security, speed and cost benefits, rather than having to choose between them.
It would also be fantastic if they separated the Spaces and CDN costs in the billing to make it clear to customers, since clearly they are separate internally.
Last but not least, they should really make this clearer in their documentation. Especially on the section that explicitly instructs how to use Signed URLs and CDN together.
TLDR: If you use signed URLs with the CDN then you'll never benefit from the Cache and pay double for the bandwidth transfer from Spaces > CDN > User rather than just Spaces > User.