This is for ad hoc generation of EPub from websites that don't have scrape well using traditional scrapers (think standard request based command line scripts or some other chrome extensions that scrape based on open tabs/window) for some reasons:
1. Usually command line scrapers and other extensions have predefined sites they work for, this one's outside of those sites
2. Or they requires nontrivial configuration and/or code
3. Some sites use javascript to dynamically generates/retrieve the text, in which case you need the browser to run the JS - This was the biggest gap for me.
4. This one runs in the browser, so maybe less likely to be detected and blocked
I also don't intend this scraper to be robust or used in a repeated fashion as a background scheduled job, that's why there's a UI for selecting key elements for scraping. It's meant to be more generalized so that you don't have to have a configuration for a site to still be able to scrape it relatively easily with just some mouse clicks.
If the site you're scraping is already handled by the other programs/extensions, then this wouldn't perform better since the other ones are specifically configured for those sites. Otherwise, this extension gives you the tool to scrape something once or twice without spending too much time coding/configuring.
I don't find myself sticking to the same site a lot, so wrote this.
Now I use it to send blogs, books and sometimes send whole archives of a website (you can use it in scripts).
You can export Kindle highlights to Obsidian, so one benefit of making these epubs is how you accumulate the highlights at one place.
Although, name is kindle-send but it can send to any ereader that uses email as a mechanism to send books.
A web scraper for blogs and mainly web novels etc and ePub parser that persists the data to database along with categories and tags, and a companion PWA for offline reading to track reading progress on various stories and let me keep multiple versions of the same story (web novels and published epub).
My local instance just runs quietly on a Synology NAS; I like not having to interact with a computer to use it. Unlike the OP, it can't be used to compile many pages/URLs into a single EPUB, though.
I once made a simple version of this concept that saves an epub file on the server‘s file system, which is then synced to my e-book reader:
https://github.com/solarkraft/webpub
The main ingredient is Postlight Parser, which gives a simplified „document“ view for a website.
open source stealth plugins don't really work now either.
you have to use real browser fingerprints.
It sort of works ie some stories just work others just get the first page.
On the one hand, they'd be adding a massive amount of free content to a platform where they make money because people pay to consume content.
On the other hand, it might actually increase sales simply because I'd spend more time using it, which would presumably result in more book purchases too.
(Also Kindle store is already full of $0 public domain stuff, so they already don't seem too bothered by that possibility.)
Are they an amazon offer or do third parties take the time to set that up?
https://sr.ht/~lioploum/offpunk/
Instead of Epub, it get catched down into text files (Gopher), Gemini files (Gemini) and HTML+images (Web Pages). You can visit the hier from ~/.cache/offpunk or directly from Offpunk.
With the "tour" function, forget about doomscrolling. You'll read all the articles in text mode sequentially until you finish down.
(dartharva's comment was the only thing here when I first looked from the front page)
Edit: Tried it with Reuters and it looks like percolate requires javascript, etc. Back to using "Print as PDF" from the browser.
If you start selling the resulting files, now that would be a copyright violation. German law has a right to create a "Privatkopie", i.e. a private copy. I guess this is similar to fair use in US law?
Where I am, it's perfectly legal.
Before cell service was as widespread as it is today, there were programs that would scrape web pages into ePUBs so you could read them later on your Palm Pilot. I used it every day during my commute. And the best part was that they ended. No mind-numbing infinite scroll.
When I switched to a "smart" phone (SonyEricsson m600c), I really missed it.
https://medium.com/@chrisdesalvo/the-future-that-everyone-fo...
I spend enough time at a computer than I shouldn't really need a smartphone outside of 'I need to message ___' or 'I need to go ___'