<@&1480181806744731748>
Following up on our last update from 3/11, where we confirmed the archive download was completed (~385TB), we made a team decision to share a bit more detail on what’s been happening so far
Since then, there’s been some great convo’s in chat between the community, mods, and devs. We figured we’d compile key points from these discussions so everyone can see what’s happening behind the scenes and understand the progress being made
📦 What’s happening right now?
Although everything has been acquired, the work is far from over.
The website is one thing, but as far as the data is concerned, The devs are focused on two major areas:
Verification and integrity checks across the entire archive
Torrent generation and distribution to make the archive accessible to everyone
The goal right now is to make sure it can be accessed in a practical way—even for those without tons of storage.
🔍 Why is it taking so long?
Some have been saying that progress has been slow and we hear ya. so here’s some perspective that was recently shared by one of the developers during a community discussion.
Working with ~385TB of data is massive. For example:
A single system hashing at ~48 MB/sec would take over 100 days nonstop to process everything
That doesn’t include additional overhead like opening compressed files and hashing their contents, which are often larger than the original archives
This is extremely computation-heavy. Good news is that Thanks to the access of high-speed infrastructure, hashing is happening at multiple GB/sec, so progress is going as fast as it can—but it will still take time.
🧪 How are files being verified?
From the recent discussion:
Hashes are compared against trusted DAT files
Crowdsourced hash contributions help validate files
Multiple hash types (including SHA-256) ensure strong verification
One question asked was: Why re-hash if hashes already exist?
Existing hashes typically apply to entire zip archives
But for proper verification against DAT files, hashes are needed for the individual files inside those archives
That means the archive has to be processed at a deeper level which, unfortunately, adds a lot more time needed for such a task. But this will ensure a much higher level of accuracy.
Of course, as with any archival or preservation project, we encourage our users to always follow standard best practices when working with downloaded files once access is available.
📁 Will the archive be updated or expanded?
We’ve also received questions about updates or including current-generation content. At this time:
No immediate plans to expand the dataset
Focus remains on verifying and stabilizing what we already have
Current-generation content is intentionally avoided to stay aligned with long-term preservation goals and minimize legal risk, ensuring the archive remains accessible over time.
🤝 Final thoughts
As many of you already know, this has been a community-driven effort from the very beginning. From the people helping source data, to those contributing hashes, to everyone asking questions and staying engaged—it all plays a role at the end of the day.
We really appreciate the patience and support from everyone while this work continues. There’s still more to be done, but progress is steady, and we’ll keep sharing updates as things move forward.
Thank you to everyone, from the bottom of our hearts, for your support. Without you, none of this would even be possible. The dedication of the crews behind the scenes, along with community contributions in helping, testing, and organizing, has been immeasurable, and we seriously can’t thank you enough. The curiosity, energy, and feedback you bring make this project what it is.
We have come so far and are getting closer every day to our goal. It’s going to be exciting once the day comes when we can open the doors for all of you.
Until then, it’s back to work for us!
Let’s get through this together,
One File at a time
❤️ — Minerva Archive Team
We also have another update on torrents beta testing. SEE BELOW
👷 🚧 Hey Everyone, 🚧 👷♀️ <@&1480181806744731748>
We got some more information regarding the torrent generation that we’ve been mentioning for quite some time now, especially in the recent mention in the last announcement.
We’ve recently begun testing on the **torrent distribution side**. For now, this is only happening between a limited group of members. These people who we like to call ‘Hoarders’ or ‘the Distribution Team’ , alongside the Development Team, are testing the torrents internally.
***(To be clear: Before you ask—as of now this is a closed beta test, we don’t have a set time for when we can give access or be able to fully release this just yet. We’ll share more details once we can be sure that it’s ready for the next steps.)***
The group is seeding and sharing the data between them as part of an “early rollout” so to speak. The plan is to start spreading the data out so it’s no longer sitting in just one place. our aim is to secure the data by having multiple copies of it so we can ensure that it goes safely, securely and consistently to those who matters most; into the hands of the community.
Basically; we’re making sure the archive stays safe and available no matter what happens. Whether that’s hardware issues, downtime, or anything unexpected that maybe we haven't thought about; if it happens, the data will always be available.
On behalf of our ongoing commitment to transparency, we are finding that we need to include more detail into our updates to, hopefully, reduce the amount of repeated questions. But for those who still need to ask, we got a couple of breakdowns for you below.
🚪 **Why isn’t it public yet?**
At this particular moment, torrent access isn’t public yet for a few reasons:
Right now, our attention is currently on making sure that whatever ideal environment we can come up with can be replicated at scale.
Things like how the torrents behave, making sure transfers go smoothly, and catching any issues early before it causes a domino effect full of problems are things that are being looked at carefully.
Keeping this limited also gives us time to figure out the best way to roll this out properly as we want to avoid opening things too early and running into instability or worse, having access unintentionally shut itself off in some way.
In short, we gotta get things as good as it can be first, so when it does open up, it’s something that works and can be as reliable as it can.
🌍 **What’s next?** 💻🖱️
While the testing happens, we also have to decide on how we are gonna get the torrents out to the community. Work on the website is being done to ensure the best rollout and availability we can safely guarantee.
As per usual, keep an eye on <#1480146718279335996> for when the torrents will be released as they will be announced there once they become available.
**Please, do not ask us to release the material in any other way other than torrents.** We know it has its shortcomings, but for now, it’s the only way we can make sure that the archive reaches everyone without falling into the same pit Myrient found itself in.
📜 **Final note** ✒️
At the end of the day, it may not seem like a lot but it’s an important step forward in making sure the archive can function as intended and pick up where Myrient will leave off.
Whatever happens, we’ll be here to work things out and deliver the best archive we can muster. But for right now, that is pretty much it. We will have more to share soon as development continues, so keep a look out for more notices from our team.
and of course, as soon as we figure out more, you guys will definitely be informed.
Thank you guys so much.
🐾 — Minerva Archive Team
In Loving Memory of MiNERVA BOT
( Please be patient -.- )
