Could be worse.
"Whether you were trying to ask a general question, or requesting the removal of your site from the Wayback Machine your data is now in the hands of some random guy. If not me, it'd be someone else."
I am starting to wonder if the chorus of 'maybe one org should not be responsible for all this; it is genuinely too important' has a point.
I agree this probably needs to be run more professionally but I think the "chorus" is missing the key fact that no one has stepped up to pay for or build an alternate and driving this one to insolvency just leaves us poorer.
I do advocate for some variant of digital prepping in my social circle, but the response has been similar to my talk about privacy. The ones that do care, have already taken steps or are in the process of some sort of solution for their use-case. Those people do not need convincing or even much help.. they mostly simply know what they want and/or need.
As for a more systemic solution.. I honestly don't know. HN itself seems pretty divided and I can absolutely understand why.
All that said, I think I agree with you. There is no real alternative now so IA needs our support at the very least until we can figure out how to preserve what was already gathered. I said the following on this forum before. Wikipedia is not hurting for money, but IA, being in legal crosshairs and whatnot, likely will.
They're providing a public service by pointing out that a massive organization controlling a lot of PII doesn't care about security at all.
I’m asking seriously - did IA do shitty things that make them a worthy cause for politically/ideologically motivated hacking?
I don't think that justifies blaming the victim here, and from what I can see the attacker doesn't seem to be motivated by anything other than funsies, but I absolutely lost a lot of faith in their leadership when they pulled the NEL nonsense. The IA is too valuable for them to act like a young activist org—there's too much for us to lose at this point. They need to hold the ground they've won and leave the activism to other organizations.
[0] https://www.wired.com/story/internet-archive-loses-hachette-...
Feeling entitled?
The discussion around IA nowadays seems a lot like random users ranting at open source maintainers in Github issues.
The black woman on the bus refusing to give up her seat was also 100% legally obviously in the wrong. IA lost not because what they were doing was morally wrong, but because each and every one of us continually refuses to agitate for the kind of change that would benefit the world.
If you want the public to have a library, you must enshrine that library's right to exist and operate in law, or it will never survive legal challenges from IP holders. Physical libraries would never be allowed to exist in modern America, not without 100 years of precedence of the first sale doctrine. You can bet your ass disney would have tried to kill such a thing. Freely watch our movies? No chance.
The IA has tried distributing their stores, but nowhere near enough people actually put their storage where their mouths are.
You can distribute less popular websites with more used ones to avoid losing it? And Torrents are good with transfering large files in my experience.
So long as this distributed protocol has the concept of individual files, there _will_ be clients out there that allow the user to select `popular-site.archive.tar.gz` and not `less-popular.tar.gz` for download.
And what one person doesn't download... they can't seed back. Distributed stuff is really good for low cost, high scale distribution of in-demand content. It's _terrible_ for long term reliability/availability, though.
I miss when TPB used to have a CSV of all their magnet links, their new UI is trash. I can't even find anything like the old days, pretty much TPB is a dying old relic.
Side note: As an outsider, and someone who hasn't tried either version of FreeNet in more than almost 2 decades, was this kind of a schism like the Python 2 vs. Python 3 kerfuffle? Is there more to it?
[0]: https://www.hyphanet.org/
[1]: https://freenet.org/
Neither version of Freenet is designed for long-term archiving of large amounts of data so it probably isn't ideally suited to replacing archive.org, but we are planning to build decentralized alternatives to services like wikipedia on top of Freenet.
[1] https://freenet.org/faq/#why-was-freenet-rearchitected-and-r...
* What is a "bird famine", and did one happen in 1880?
* Did any astrologer ever claim that the constellations "remember" the areas of the sky, and hence zodiac signs, that they belonged to in ancient times before precession shifted them around?
* Who first said "psychology is pulling habits out of rats", and in what context? (That one's on Wikiquote now, but only because I put it there after research on IA.)
Or consider the recently rediscovered Bram Stoker short story. That was found in an actual library, but only because the library kept copies of old Irish newspapers instead of lining cupboards with them.
The necessary documents to answer highly specific questions are very boring, and nobody has any reason to like them.
Sort of like the bittorrent algorithm that favors retrieving and sharing the least-available chunks if you haven't assigned any priority to certain parts.
* Strictly speaking, running in-browser, but that sounded like "Bowser" so I wrote online instead.
Including the index itself? That would be awesome.
In practice, that's mostly how they're being used.
But the protocol does support mutation. The BEP describing the behavior even has archive.org as an example...
> The intention is to allow publishers to serve content that might change over time in a more decentralized fashion. Consumers interested in the publisher's content only need to know their public key + optional salt. For instance, entities like Archive.org could publish their database dumps, and benefit from not having to maintain a central HTTP feed server to notify consumers about updates.
What there isn't is a currently maintained and advertised client and plan. That I can find. Clunky or not, incomplete or not.
There are other systems that have a rough plan for duplication and local copy and backup. You can easily contribute to them, run them, or make local copies. But not IA. (I mean you can try and cook up your own duplication method. And you can use a personal solution to mirror locally everything you visit and such.) No duplication or backup client or plan. No sister mirrored institution that you might fund. Nothing.
Typically because most people who have the upload, don't know that they can. And if they come to the notion on their own, they won't know how.
If they put the notion to a search engine, the keywords they come up with probably don't return the needed ELI5 page.
As in: How do I [?] for the Internet Archive?, most folks won't know what [?] needs to be.
Most casual visitors to IA don't know that. Which is the point.
Giving up is for others.
Then rename it from "torrent" to something else.
The bad reputation is inherent to the tech, not a random quirk.
Downloading from example.com is just peer to peer with someone big. There's lots of hosting providers and DNS providers that are happy to host illegal-in-some-places content.
The protocols for downloading from example.com are assymettrical client server architectures, not symmetrical decentralized peer to peer.
But anyways not the case for the wayback product which is the unique core to IA.
Well, OK, maybe other webpage archives don't work as well, I haven't tried them, but there are others. And they're newer, so don't have such extensive historical pages.
Large numbers of Wikipedia references (which relied on IA to prevent link rot) must be completely broken now.
If this is what people think we need to work on education...
The average person, in my experience, can barely work a non-cellphone filesystem and actively stresses when a terminal is in front of them, especially for a brief moment. Education went out the window a decade ago.
Criminals using tools does not make the tools criminal.
This has precedent in illegal drug categorization, it's not just about the damage, but its ratio of noxious to helpful use.
Societies should criminalize behavior and then (shocker!) enforce the laws! Let tools be tools.
What are some legal torrent trackers?
Depends on the jurisdiction. Remember what happened in the The Pirate Bay trial?
To me that's not even related to it being a torrent tracker, just that they were "aiding and abetting" copyright infringement.
In Law the technicalities matter.
Trackers generally do not host any content, just hashcodes and (sometimes) meta data descriptions of content.
If "your" (ie let's say _you_ TZubiri) client is distributing child pornography content because you have a partially downloaded CP file then that's on _you_ and not on the tracker.
The "tracker" has unique hashcode signatures of tens of millions of torrents - it literaly just puts clients (such as the one that you might be running yourself on your machine in the example above) in touch with other clients who are "just asking" about the same unique hashcode signature.
Some tracker affiliated websites (eg: TPB) might host searchable indexes of metadata associated with specific torrents (and still not host the torrents themselves) but "pure" trackers can literally operate with zero knowledge of any content - just arrange handshakes between clients looking for matching hashes - whether that's UbuntuLatest or DonkeyNotKong
On the other hand I also believe that a tracker that hosts hashes of illegal content, provides search facilities for and facilitates their download, is responsible, in a big way. That's my personal opinion and I think it's backed in cases like the pirate bay and sci hub.
That 0 knowledge tracker is interesting, my first reaction is that it's going to end up in very nasty places like Tor, onion, etc..
Most actual trackers are zero knowledge.
A tracker (bit of central software that handles 100+ thousand connections/second) is not a "torrent site" such as TPB, EZTV, etc.
A tracker handshakes torrent clients and introduces peers to each other, it has no idea nor needs an idea that "SomeName 1080p DSPN" maps to D23F5C5AAE3D5C361476108C97557F200327718A
All it needs is to store IP addresses that are interested in that hash and to pass handfuls of interested IP addresses to other interested parties (and some other bookkeeping).
From an actual tracker PoV the content is irrelevant and there's no means of telling one thing from another other than size - it's how trackers have operated for 20+ years now.
Here are some actual tracker addresses and ports
udp://tracker.opentrackr.org:1337/announce
udp://p4p.arenabg.com:1337/announce
udp://tracker.torrent.eu.org:451/announce
udp://tracker.dler.org:6969/announce
udp://open.stealth.si:80/announce
udp://ipv4.tracker.harry.lu:80/announce
https://opentracker.i2p.rocks:443/announce
Trackers can hand out .torrent files if asked (bencoded dictionaries that describe filenames, sizes, checksums, directory structures of a torrents contents) but they don't have to; mostly they hand out peer lists of other clients .. peers can also answer requests for .torrent files.
A .torrent file isn't enough to determine illegal content.
Pornography can be contained in files labelled "BeautifulSunset.mkv" and Rick Astley parody videos can frequently be found in files labelled "DirtyFilthyRepubicanFootTappingNudeAfrica.avi"
Given that it's not clear how trackers could effectively filter by content that never actually traverses their servers.
Mathematically a tracker would offer a function that given a hash, it returns you a list of peers with that file.
While a "torrent site" like TPB or SH, would offer a search mechanism, whereby they would host an index, content hashes and english descriptors, along with a search engine.
A user would then need to first use the "torrent site" to enter their search terms, and find the hash, then they would need to give the hash to a tracker, which would return the list of peers?
Is that right?
In any case, each party in the transaction shares liability. If we were analyzing a drug case or a people trafficking case, each distributor, wholesaler or retailer would bear liability and face criminal charges. A legal defense of the type "I just connected buyers with sellers I never exchanged the drug" would not have much chance of succeding, although it is a common method to obstruct justice by complicating evidence gathering. (One member collects the money, the other gives the drugs.)
> Is that right?
More or less.
> In any case, each party in the transaction shares liability.
That's exactly right Bob. Just as a telephone exchange shares liability for connecting drug sellers to drug buyers when given a phone number.
Clearly the telephone exchange should know by the number that the parties intend to discuss sharing child pornography rather than public access to free to air documentaries.
How do you propose that a telephone exchange vet phone numbers to ensure drugs are not discussed?
Bear in mind that in the case of a tracker the 'call' is NOT routed through the exchange.
With a proper telephone exchange the call data (voices) pass through the exchange equipment, with a tracker no actual file content passes through the trackers hardware.
The tracker, given a number, tells interested parties about each other .. they then talk directly to each other; be it about The Sky at Night -s2024e07- 2024-10-07 Question Time or about Debbie Does Donkeys.
Also keep in mind that trackers juggle a vast volume of connections of which a very small amount would be (say) child abuse related.
I'll restate the principle of good usage to bad usage ratio, telephone providers are a well established service with millions of legitimate users and uses. Furthermore they are a recognized service in law, they are regulated, and they can comply with law enforcement.
They are closer to the ISP, which according to my theory has some liability as well.
It's just a matter of the liability being small and the service to society being useful and necessary.
To take a spin to a similar but newer tech, consider crypto. My position is that its legality and liability for illegal usage of users (considering that of exchanges and online wallets, since the network is often not a legal entity) will depend on the ratio of legitimate to ilegitimate use that will be given to it.
There's definitely a second system effect, were undesirables go to the second system, so it might be a semantical difference unrelated to the technical protocols. Maybe if one system came first, or if by chance it were the most popular, the tables would be turned.
But I feel more strongly that there's design features that make law compliance, traceability and accountability difficult. In the case of trackers perhaps the microservice/object is a simple key-value store, but it is semantically associated with other protocols which have 'noxious' features described above AND are semantically associates with illegal material.
Ditto trackers.
Have a look at the graphs here: https://opentrackr.org/
Over 10 million torrents tracked daily, on the order of 300 thousand connections per second, handshaking between some 200 million peers per week.
That's material from the Internet Archive, software releases, pooled filesharing, legitimate content sharing via embedded clients that use torrents to share load, and a lot of TV and movies that have variable copyright status
( One of the largest TV|Movie sharing sites for decades recent closed down after the sole operator stopped bearing the cost and didn't want to take on dubious revenue sources; that was housed in a country that had no copyright agreements with the US or UK and was entirely legal on its home soil.
Another "club" MVGroup only rip documentaries that are "free to air" in the US, the UK, Japan, Australia, etc. and in 20 years of publicaly sharing publicaly funded content haven't had any real issues )
> the ISP, which according to my theory has some liability as well.
The world's a big place.
The US MPA (Motion Picture Association - the big five) backed an Australian mini-me group AFACT (Australian Federation Against Copyright Theft) to establish ISP liability in a G20 country as a beach head bit of legislation.
That did not go well: Roadshow Films Pty Ltd v iiNet Ltd decided in the High Court of Australia (2012) https://en.wikipedia.org/wiki/Roadshow_Films_Pty_Ltd_v_iiNet...
The alliance of 34 companies unsuccessfully claimed that iiNet authorised primary copyright infringement by failing to take reasonable steps to prevent its customers from downloading and sharing infringing copies of films and television programs using BitTorrent.
The trial court delivered judgment on 4 February 2010, dismissing the application and awarding costs to iiNet.
An appeal to the Full Court of the Federal Court was dismissed.
A subsequent appeal to the High Court was unanimously dismissed on 20 April 2012.
This case is important in copyright law of Australia because it tests copyright law changes required in the Australia–United States Free Trade Agreement, and set a precedent for future law suits about the responsibility of Australian Internet service providers with regards to copyright infringement via their services.
> but it is semantically associated with other protocols which have 'noxious' features described above AND are semantically associates with illegal material.
Gosh, semantics hey. Some people feel in their waters that this is a protocol used by criminals and must therefore by banned or policed into non existance?
Is that a legal argument?
I also indicated above that having knowledge of .torrent manifests is problematic as that doesn't provide real actual knowledge of file contents just knowledge of file names ... LatestActionMovie.mkv might be a rootkit virus and HappyBunnyRabbits.avi might be the worst most exploitative underage pornography you can think of.
Some trackers are also private and require membership keys to access.
I was skating a lot as TZubiri seems unaware of many of the actual details and legitimate use cases, existing law, etc.
This is a brilliant system relying on a randomised consensus protocol. I wanted to do my info sec dissertation on it, but its security model is extremely well thought out. There wasn't anything I felt I could add to it.
For a large-scale archival project, it might not be ideal. Maybe something based on erasure coding would be better. Do you know how LOCKSS compares?
Was that any file in particular? I just tried it myself with a 257mb PDF (as reported by `ls -lrth`) and doesn't seem to add that much overhead:
$ du -sh ~/.ipfs
84K /home/user/.ipfs
$ ipfs add ~/Downloads/large\ PDF\ File.pdf
added QmSvbEgCuRNZpkKyQm6nA5vz5RTHW1nxb6MJdR4cZUrnDj large PDF File.pdf
256.58 MiB / 256.58 MiB [============] 100.00%
$ du -sh ~/.ipfs
264M /home/user/.ipfsThe design is really very good.
If different data always gets a different reference, it's easy to know if you have enough backups of it. If the same name gets you a pile of snapshots taken under different conditions, it's hard to be sure which of those are the thing that we'd want to back up for that particular name.
https://github.com/internetarchive/dweb-archive/blob/master/...
(this doc is 5-6 years old though, and I'm not sure what may have changed since then)
In my own (toy-scale) IPFS experiments a couple years ago it has been rather usable, but also the software has been utterly insane for operators and users, and if I were IA I would only consider it if I budgeted for a from-scratch rewrite (of the stuff in use). Nearly uncontrollable and unintrospectable and high resource use for no apparent reason.
What's the point of using IPFS then? Others can still spread the file elsewhere and verify it's the correct one, by using the exact same ID of the file, although on two different networks. The beauty of content-addressing I guess.
Especially if it's about having an Internet Archive backup.
Would be people be willing to buy an IA box that hosted a shard of random content along with the things they wanted themselves?
Unfortunately, when I talked to a few archival teams (including the IA) about whether they'd be interested in using it, I either got no response or a negative one.
If you have a raid, then you have 2 copies with like 99.99% availability and 5 mean time years to failure.
With a volunteer drive you have like ?% availability and ?% years to failure? You can't depend on it.
Also the average value of data is very low, you don't want to be making many copies of for no reason.
> Also the average value of data is very low, you don't want to be making many copies of for no reason.
The reason is that the value of that data is high to the archivist, since they want to preserve it.
Realistically you won't get enough volunteer-storage to cover one IA. And even if you did, it wouldn't satisfy the mission requirements, which is to store reliably for decades all of the data.
https://docs.google.com/document/d/1qKgIjUTef-I-BLWjn4sEIbYo...
I'll write up a more detailed article on it, though, it'll be good to at least have the doc public somewhere.
There are so many proven distributed archiving systems, a lot of which are mentioned in these comments.
As for technical attacks, I'm not an expert but I'd assume it's more difficult for bad actors to bring down decentralized networks. Has the BitTorrent network ever gone offline because it was hacked for example? That seems like it would be extremely hard to do, not even the movie industry managed to take them down.
With the 30-second "time to first byte" speed we all know and love from IA, I'm pretty sure it'd only get faster when you're the only person accessing an obscure document on a random person's shoebox in Korea as compared to trying to fetch it from a centralised server that has a few thousand other clients to attend to simultaneously
Depending on scale that’s not necessarily true. I find even today there are many services that cannot keep up with my residential fiber connection (3Gbps symmetrical), whereas torrents frequently can. IA in particular is notoriously slow when downloading from their servers, and even taking into account DHT time torrents can be much faster.
Now if all of their PBs of data were cached in a CDN, yeah that’s probably faster than any decentralized solution. But that will take a heck of a lot more money to maintain than I think is possible for IA.
Risk management is a balance, not fearmongering as you say. That's why I'd rather use advice from people with daily experience than look at the newsworthy experiences ("nothing happened today, again; regular security patches working fine" you'll never see) and conclude you'd attract threats and cyber attacks just by hosting backup copies of parts of the Internet Archive
https://news.ycombinator.com/item?id=41860909
I'd never heard of it, but their responses to question and comments in that thread were really really good (and I now have "install and configure archivebox on the media server" on my upcoming weekend projects list).
Right now there are torrents and I do keep any torrents I download from IA in my client for years but torrents means I only get to contribute by sharing the things I downloaded in the past.
I was looking into using R2 as a web seed for the torrent but I don't _really_ want to spend much to upload content that is going to get "stolen" and reuploaded by content farms anyway you know?
Centralized entities emerge to absorb costs because nobody else can do it as efficiently alone.
>What happens when someone storing decentralized data decides to exit?
They exit, and they no longer store decentralized data. At the very least, IA would still have their copy(s), and that data can be spread to other decentralized nodes once it has been determined (through timeouts, etc) that the person has exited.
> Will data be copied to multiple places[...]?
Ideally, yes. It is fairly trivial to determine the reliability of each member (uptime + hash checks), and reliable members (a few nines of uptime and hash matches) can be trusted to store data with fewer copies while unreliable members can store data with more copies. Could also balance that idea with data that's in higher demand, by storing hot data lots of times on less reliable members while storing cold data on more reliable members.
> who pays for the decentralized storage long term? [...] who is going to pay for doubling, tripling or more the storage costs for backups?
This is unanswered for pretty much any decentralized storage project, and is probably the only important question left. There are people who would likely contribute to some degree without a financial incentive, but ideally there would be some sort of reward. This in theory could be a good use for crypto, but I'd be concerned about the possible perverse incentives and the general disdain the average person has for crypto these days. Funding in general could come from donations received by IA, whatever excess they have beyond their operating costs and reserve requirements - likely would be nowhere near enough to make something like this "financially viable" (i.e. profitable) but it might be enough to convince people who were on the fence to chip in few hundred GB and some bandwidth. This is an open question though, and probably the main reason no decentralized storage project has really taken off.
History has always gotten rewritten throughout time. If you have a giant library it's easier for bad actors to gain influence and alter certain books, or remove them. This isn't just theoretical, under external pressure IA has already removed sites from its archive for copyright and political reasons.
There are also threats that are generally not even considered because they happen with rare frequency, but when they happen they're devastating. The library of Alexandria was burned by Julius Caesar during a war. Likewise, if all your servers are in one country that geographic risk, they can get destroyed in the event of a war or such. No one expects this to happen today in the US, but archives should be robust long term, for decades, ideally even centuries.
I would wager at least 95% of "digital memory" archived is just absolute garbage from SEO spam to just some small websites holding no actual value.
The true digital memory of the world is almost entirely behind the walls of reddit, twitter, facebook, and very few other sites. The internet landscape has changed massively from the 90s and 2000s.
We are talking about an (almost) worldwide archive after all.
Nobody has ever stopped a competitive alternative from existing. Feel free to give it a shot. You have a head start with all the work that they've done and shared.
The sad reality is that a lot of people are unfairly attacked on the internet and many go unpunished due to lack of investigative focus, resources, etc.
Those who don't get the salt shaker bit, here's the original of the ancient wisdom:
https://web.archive.org/web/20060619131835/http://xelios.liv...
Choose any translation:
https://malaya-zemlya.livejournal.com/697779.html
https://personal-view.com/talks/discussion/25915/humor-hacke...
https://www.linkedin.com/pulse/hacker-restaurant-alexander-s...
What gives that impression? Everything I've seen about the attacker's messaging says "vandal punk(s)" to me, and nothing in what I've seen of the IA's systems screams Fort Knox. It wouldn't surprise me if they actually had a pretty lax approach to security on the assumption that there's very little reason to target them.
I think you should draw your own more informed conclusions, but it smells a lot like feds to me.
Oh how wrong you are.
There is nothing boring in a target that can be used to validate others' lies and potential hypocrisy, changes in their policies etc.
The wayback machine itself serves as a truly priceless way to go back through someone's public life.
This is also right before a major U.S. election, and that might not be a coincidence. Someone might be trying to get Trump elected by drawing attention to the Israel-Gaza conflict, a topic that isn't exactly a winning issue for Harris.
That's just one possibility of many though. I mean, might just be regular assholes that happen to also be pro-Palestinian. Or pro-Hamas, which isn't necessarily the same thing.
I mean it's where we go to prove a politician said something after they deleted it or where a government changes the wording of something...
I'd argue it's one of the juicer political targets if you're actually wanting to do something.
The breach happened over a week before the DDoS attack, according to Troy Hunt.
Stop looking for conspiracy theories.
There's just too much "means, motive and opportunity" there.
If the state of a webpage in the past matters to you, you need a record that won't cease to exist when your opposition asks it to. This is the concept behind perma.cc.
edit: "Other types of removal requests may also be sent to info@archive.org. Please provide as clear an explanation as possible as to what you are requesting be removed for us to better understand your reason for making the request.", https://help.archive.org/help/how-do-i-request-to-remove-som...
I suspect they DO delete some things.
If you were photocopying a textbook and giving it to your classmates, the publisher could have their lawyer send you a Cease and Desist letter telling you to stop (or else). But if they told you to burn your copy of the textbook then they would be overreaching, and everyone would laugh at them when you took that story to the papers.
Legal reasoning from made‐up examples is generally a bad idea, but I think you can safely reason from that one.
I’m not privy to the actual communications in these cases, but I suspect that instead of replying back with “we deleted the content from the Archive”, they instead say something anodyne like “the content is no longer available via the Wayback Machine”. Smart lawyers will notice the difference, but then a smart lawyer wouldn’t have expected anything else.
You are wrong, copyright specifically prohibits copying, not distribution. They can get a cease and desist that requests you destroy property and they ca get a court order backing that which will put you into contempt of court if you fail to do so.
Proving damages is easier with distribution, but that is a civil matter not a criminal matter.
You do realise the "downloading" is implicitly a copy.
If you want to actually have a civil discussion then you need to make some reasonable argument than "They're not making copies either."
Sounds like whatever role you played at IA when you were there didn't give you any actual insight into what happens in operation and you simply tried to prove your point with an appeal to authority instead of backing it with facts and reason.
What? That's the only way to do legal reasoning, and as an obvious consequence it's how both lawyers and judges do it.
Even better would be to quote from some case where a judge has applied the law to actual events.
My intuition would say there are 3 cases when content ceases to become available at the original site:
- The host becomes unable to host the content for some reason (bankruptcy, death, etc.) in which case I assume the archive persists.
- The host is externally required to remove the content (copyright, etc.) in which case I assume IA would face the same external pressure? But I’m not sure on that.
- The host/owner has a change of heart about publishing the content. This borders more on IA acting as reputation management on the part of the original host/owner. Personally I think this is hardest to defend but also probably the least common case. In this case I’d think it’s most often to hide something the original host doesn’t want the public finding out later, but that also seems to make it more valuable to be publicly available in the archive. Plus, from a historian/journalist perspective, it’s valuable to be able to track how things change over time, and hiding this from the public prevents that. Though to be honest I’m kind of in two minds here because on the other hand I’m generally of the opinion that people can grow and change, and we shouldn’t hold people to account for opinions they published a decade ago, for example. I’m also generally in favor of the right to be forgotten.
Would appreciate your thoughts here.
When IA shows you what a website looked like in the past, they are reproducing a copyrighted work and distributing it to you. In some cases, perhaps many, this is fair use. IA cannot really know ahead of time which viewers would be exercising their fair use rights and which would not. Instead, IA just makes everything available without trying to guess whether the access would fall under fair use or not. That means that many times, possibly most of the time, IA is technically breaking the law by illegally distributing copies of copyrighted works.
But _owning_ a copy of a copyrighted work is never prohibited by copyright. It doesn’t matter how you got the copy either.
Therefore, pretty much any time someone asks for something to be hidden or removed on copyright grounds, they go ahead and hide it. They don’t bother to delete it though, because copyright doesn’t require them to. If a copyright holder asks for it to be deleted then they are overreaching, and should know that any sane person would object. But as far as I am aware IA doesn’t actually bother to object in writing; they just hide the content and move on.
This means that researchers can visit the archive in person and request permission to see those copies. For example if you are studying the history of artistic techniques in video games using emulated software on IA, you might eventually notice that all the games from one major publisher are missing (except iirc the original Donkey Kong, because they don’t actually own the copyright on that one). You could then journey to the Archive in person to see the missing material and fill in the gaps in your history. Or you could just ignore them entirely out of spite. This is no different than viewing rare books held by any library, or viewing unexhibited artifacts held by a museum, etc
It’s a shame that to be able to run an above-board _Internet_ Archive one needs to bend to the whim of anachronistic copyright law and forego all the benefits of the internet in the first place. This seems like it would inevitably mean that any _internet_ archive that is truly accessible over the _internet_ would be forced to operate illegally in a similar manner to SciHub.
I know I hold a rather strong opinion regarding copyright law (I’m not looking to debate it here as I know others hold different opinions which is totally fine), but IMHO copyright law has been a major blight on humanity at large and especially the internet. Major reform is in order at the very least, if not total abolishment.
* it prevents victims from performing discovery (gathering evidence) before starting a trial or confiding to an expensive lawyer whose loyalty may turn out to systematically lie with the perpetrators or highest bidders.
* it prevents people who requested a snapshot (and thus know a specific URL with relevant knowledge) from proving their version of events to acquaintances, say during or after a court case in the event their lawyers just spin a random story instead of submitting the evidence as requested, since disloyal lawyer will have informed counterparty and counterparties will have requested "removal" of the page at IA, resulting in psychological isolation of the victim since victim can no longer point to the pages with direct and or supporting evidence.
Anyone with even basic understanding of cryptographic hashes and signatures would understand that:
1) for a tech-savvy entity (which an internet archival entity automatically is expected to be)
2) in the face changing norms and values (regardless of static or changing laws: throughout history violations were systematically turned a blind eye to)
3) given the shameless nature of certain entities, self-describing their criminal behavior on their commercial webpages
Any person understanding above 3 points concludes that such an archival company can impossibly assume some imaginary "middle ground" between:
A) Defender of truth and evidence, freedom fighter, human rights activist, so that humanity can learn from mistakes and crimes
or
B) Status quomplicit opressor of evidence
Because any imaginary hypothetical "middle ground" entity would quickly be inundated by legal requests for companies hiding their suddenly permanently visible crimes, and simultaneously for reinstantiations by victims pleading public access to the evidence.
Once we know its either A or B, and recalling point "tech savvy" (point 1), we can summarily conclude that a class A archival entity would helpfully assist victims as follows: don't just provide easy page archival buttons, but also provide attestations: say zip files of the pages, with an autogenerated legalese PDF, with hashes of the page and the date of observation, cryptographically signed by the IA. This way a victim can prove to police, lawyers, judges, or in case those locally work against them, prove to friends, family, ... that the IA did in fact see the information and evidence.
I leave it to the reader to locate the attestation package zips for these pages, in order to ascertain that the IA is a class A organization, and not a class B one.
I haven't read this paper yet, but...
https://www.tesble.com/10.1080/0270319x.2021.1886785
from the abstract:
> The article concludes that Perma.cc's archival use is neither firmly grounded in existing fair use nor library exemptions; that Perma.cc, its "registrar" library, institutional affiliates, and its contributors have some (at least theoretical) exposure to risk
It seems that the article is about copyright, but of course there are several other reasons that might justify takedown of content stored on perma.cc:
- Right to be forgotten... perma.cc might be able to ignore it, but could this lead to perma.cc being blocked by european ISPs
- ITAR stuff
- content published by entities recognized by $GOVERNMENT as terrorist organizations
- revenge porn
- CSAM
I'll hold my breath.
This is in line with its mission as the "Library of Congress". Being able to have an accurate record of what was on the Internet at a specific point in time would be helpful when discussing legislation or potential regulation involving the internet.
[0] https://www.loc.gov/programs/web-archiving/about-this-progra...
[1] https://blogs.loc.gov/thesignal/2023/08/the-web-archiving-te...
[2] https://en.m.wikipedia.org/wiki/List_of_Web_archiving_initia...
You basically have to re-perimeterize your topology with known good working security, and re-examine trusted relationships starting with a core group of servers and services, and then expanding outwards, ensuring proper segmentation along the way. Its a lot easier with validated zero trust configurations, but even then its a real pain (especially when there is a hidden flaw in your zero-trust config somewhere) and its very heavy on labor. Servers and services also need to ensure they have not deviated from their initial known desired states.
Some bad guys set traps in the data/services as timebombs, that either cross-polinate, or re-compromise later. There are quite a lot of malicious ****s out there.
Even if it's not publicly available...
Do you know Nanowar? They began there.
Also, as commercal music has been deliberately dumbed down for the masses (in paper, not by cheap talking), discovering Jamendo and Magnatune in late 00's has been like crossing a parallel universe.
This is quite embarrassing. One of the first things you do when breached at this level is to rotate your keys. I seriously hope that they make some systemic changes, it seems that there were a variety of different bad security practices.
I'd say they need support. They didn't abandon or pervert their mission, they relied on people they trusted who weren't equipped to also handle security. If your house were broken into, I wouldn't start a neighborhood petition for you to move out, because you didn't cause it.
They may be in a rut, but short of you or someone else building an IA replacement that settles all of your concerns and commiting to it for twenty five years with no serious compromises, you're probably punching a little above your weight on the topic.
I'm curious what other information on that site you think was valuable to have available to the general public? Nothing has been lost in terms of historical data, it's only the immediate disemmination that has been slowed.
I'm really trying to understand why I should disagree with the IA's choice here. The IA is an archival service, not a distribution platform and it is not their job to help you distribute content that other people find objectionable. Their job is to make and keep an archive of internet content so that we don't lose the historical record. Blocking unrestricted public access to some of that content doesn't harm that mission and can even support it.
kiwifarms could spin up their own infrastructure, serve their own content for the world, but it turns out technology is a social problem more than a technical problem.
anyone that wants to stand up and be the digital backbone of “kiwi farms” can, but only the internet archive gets flack for not volunteering to be the literal kiwi farm.
for example, the pirate bay goes offline all the time, but it turns out the people that use it, care enough to keep it online themselves.
Website caches can be handled differently, but bulk collection of commercial works can't have this same public access treatment. It's crazy to think this wouldn't be a huge liability.
Battling for copyright changes is valiant, but orthogonal. And the IA by trying to do both puts its main charter--archival--at risk.
The IA should let some other entity fight for copyright changes.
I say this as an IA proponent and donor.
What makes you feel entitled to the content of the publisher before the copyright expires? Do you feel that you deserve access to everything because you've deemed the concept of ownership around book publishing immoral?
You can't just take a digital copy of a physical book and give it to everyone worldwide. That isn't your choice or decision to make nor is it ethical to ascribe malice to simply retaining distribution rights to content they own.
"Make publishers richer", it's actually just honoring the concept of ownership...
If publishers didn’t engage in tactics like “library pricing” and preventing people from actually purchasing the books, I might feel differently. Right now, I see this archiving stuff as a Robin Hood story (which fwiw, every version of this story you may have seen/heard is probably still copyrighted) and I hope the publishers die or are replaced.
Internet Archive should focus on its mission of archival. Let other groups figure out copyright.
By taking on both tasks, IA risks everything and could stumble in its goal to be an archivist platform. We need an entity dedicated to recording history. IA is that. They're just biting off way too much to chew and making powerful enemies along the way.
By making the archive public, sure, we have a bit of a "piracy" issue. However, we can also verify they are actually archiving the things they say they are, point out mistakes, and ask them to remove things from the archive.
And it doesn't.
IA should collect these materials, but they shouldn't be playing fast and loose by letting everyone have access to them. That's essentially providing the same services as the Pirate Bay under the guise of archivism.
This puts IA at extreme legal risk. Their mission is too important to play such games.
Which means no one alive today would ever be able to see them out of copyright. It also requires an unfounded belief that major copyright owning companies won't extend copyright lengths beyond current lengths which are effectively "forever".
These people are not dispirited whatsoever, if anything they are half-cocked that these script kiddies found an easy target.
They could have done much worse but they chose not to and instead made it public. Which state actor does that?
Washington Post: The organization has “industry standard” security systems, Kahle said, but he added that, until this year, the group had largely stayed out of the crosshairs of cybercriminals. Kahle said he’d opted not to prioritize additional investments in cybersecurity out of the Internet Archive’s limited budget of around $20 million to $30 million a year.
Security by its very nature has a problem of knowing when to stop. There's always better security for an ever increasing amount of money and companies don't sign off on budgets of infinity dollars and projects of indefinite length. If you want security at all you have bound the cost and have well-defined stopping points.
And since 5 security experts in a room will have 10 different opinions on what those stopping points should be— what constitutes "good-enough" they only become meaningful when there's industry wide agreement on them.
The budget that it takes to protect against a script kiddy is a tiny fraction of the budget it takes to protect from a professional hacker group, which is a fraction of what it takes to protect from nation state-funded trolls. You can correctly decide that your security is “good enough” one day, but all it takes is a single random news story or internet comment to put a target on your back from someone more powerful, and suddenly that “good enough” isn’t good enough anymore.
The Internet Archive might have been making the correct decision all this time to invest in things that further its mission rather than burning extra money on security, and it seems their security for a long time was “good enough”… until it wasn’t.
If that happens you need to seriously rethink your hiring process.
We can’t all have the latest EPYC processors with the latest bug fixes using Secure Enclaves and homomorphic encryption for processing user data while using remote attestation of code running within multiple layers of virtualization. With, of course, that code also being written in Rust, running on a certified microkernel, and only updatable when at least 4 of 6 programmers, 1 from each continent, unite their signing keys stored on HSMs to sign the next release. All of that code is open source, by the way, and has a ratio of 10 auditors per programmer with 100% code coverage and 0 external dependencies.
Then watch as a kid fakes a subpoena using a hacked police account and your lawyers, who receive dozens every day, fall for it.
They sell paid services to universities and governments, so downtime isn't a great look either.
> it's not a bank
They tried that too. Didn't go well.
https://ncua.gov/newsroom/press-release/2016/internet-archiv...
That's incorrect IMHO: You are describing outcomes; practices are about procedures. In particular, necessary to the understanding and use of best practices is that do not guarantee outcomes.
Any serious management balances risks, which includes the inevitability, though unpredictable, of negative outcomes. It's impossible to prevent them - not NASA, airlines, surgeons, etc, can prevent them all, and they accept that.
It's a waste of resources to spend more preventing them than you lose overall. Best practices do not provide perfect outcomes; they provide the most reduced trade-offs in risk and cost.
Despite all of the positive self-talk, I don't know if they realize how important they are, or how easy it would be for them to find good help and advice if their management were transparent and everything was debated in public. That may have protected it to some extent; as a counterexample, Wikipedia has been extremely fragile due to its transparency and accessibility to everyone. With IA being driven by its creator's ideology, maybe that ideology should be formalized and set in stone as bylaws, and the torch passed to people openly debating how IA should be run, its operations, and what it should be taking on.
I don't mean they should be run by the random set of Confucian-style libertarian aphorisms that is running the credibility of Wikipedia into the ground, but Debian is a good model to follow. Or maybe do better than both?
While I have no idea how Debian is actually funded I'd agree. One issue might be that The Internet Archive actually need to have people on staff, not sure if Debian has that requirement. You're not going to get people to man scanner or VHS players 8 hours a day without pay, at least not at this scale.
The Internet Archive needs a better funding strategy that asking for money on their own site. People aren't visiting them frequently enough for that to work. They need a fundraising team, and a good one.
Finding managers are probably even worse. They can't get a normal CEO type person, because they aren't a company and the type of people who apply to or are attracted to running non-profit, server the community, don't be evil organisation are frequently bat-shit crazy.
Sadly, SQlite is the only software organization I know of that has this spirit.
I appreciate their ethos and I've used the site many times (and donated!), but clearly it's at the point where Kahle et al just aren't equipped either personally (as a matter of technical expertise) or collectively (they are just a handful of people) to be dealing with what are probably in many cases nation-state attacks. Kahle's attitude towards (and misunderstanding of) copyright law is IMO proof that he shouldn't be running things, because his legal gambles (gambles that a first year law student could have predicted would fail spectacularly) have put IA at long term risk (see: Napster). And this information coming out over the past few weeks about their technical incompetence is arguably worse, because the tech side of things are what he and his team are actually supposed to be good at.
It's true that Google and Microsoft and others should be propping up the IA financially but that isn't going to solve the IA's lack of technical expertise or its delusional hippie ethos.
Can you elaborate? I'm aware of Wikipedia having very particular rules and lots of very territorial editors, but I'm not sure how this runs their credibility into the ground aside from pissing off the far right when they come in with an agenda to push.
With everything that’s going on, it’s highly suspicious that this is happening right after they upset some very rich rent seekers.
Who are "they"? And who are the "very rich rent seekers"?
Absolutely moronic and unbased implication. The “rent-seekers” won their case and have zero interest in being implicated in dumb palace-intrigue style hacking. I mean, fuck those guys, but to bring up allegations like that is big stupid.
That makes no sense.
The fact that they won their case gives even greater cause in ensuring that what they want goes through. Doesn't mean they have to be classy about it, or that Internet-based means of sabotage are impossible implications (given that the IA literally is about putting things up on the Internet that some want to be taken down).
Which is why they will continue their attack through the court system until they get everything they want, up to and including shutting down the archive for good. There's zero reason for them to risk being implicated in a crime when their opponent is already down for the count.
Most of us care mainly about the Wayback Machine and archiving webpages; not borrowing books still under copyright and fighting publishers.
Under discovery in the case, it turned out that Internet Archive didn't keep accurate records of what they loaned out either. Another example of sloppy engineering that directly impacts their core mission.
The fate of the organization now rests on the outcome of other lawsuits. In one, Internet Archive argues that they are allowed to digitize and publish Frank Sinatra records because the pops and crackles on them makes it Fair Use.
If they did all this cleanly under a different LLC, I'd sit back and enjoy the show. But they didn't.
(Hot Fuzz reference. https://www.youtube.com/watch?v=oQzrR6nOkYg )