View Issue Details

IDProjectCategoryView StatusLast Update
0000067FFDLBugpublic2020-12-28 22:50
Reporterabzb13 Assigned ToRaimond  
PrioritynormalSeveritymajorReproducibilitysometimes
Status assignedResolutionopen 
PlatformMicrosoftOSWindowsOS Version10
Summary0000067: (0.9.2) Long fics get cut off
DescriptionFics with many chapters have their latter portions of chapters cut off (after a point, each chapter downloads as the chapter heading and a blank page).

I have a feeling that it is looking for rapid requests in sequence and blocking them after some threshold is reached.

Could an option to pull longer fics slowly prevent this from happening?

I'd happily live with waiting several minutes for a long fic to download in exchange for it getting there eventually :)
Steps To ReproduceA couple of fics this happened with:

https://www.fanfiction.net/s/12515214/
https://www.fanfiction.net/s/12622310/
Additional InformationI have attached a screencap of one of the blank chapters (for what little that's worth)

I've never posted here before, so thank you for the awesome program!!
TagsNo tags attached.

Activities

abzb13

2020-12-22 19:37

reporter  

hexarisius

2020-12-23 15:44

reporter   ~0000059

Hi,

First of all thanks for the program :D and have a Merry Christmas soon.

I have the same problem with the chapters. They appear blanco. Sometimes I do see some text other times it's completely blanco

https://www.fanfiction.net/s/13287185/8/Harry-Potter-and-the-Lightning-Child
https://www.fanfiction.net/s/13652455/1/Harold-Wayne
https://www.fanfiction.net/s/13456617/1/Dark-Lord-Potter

Sincerely Eleonor

Raimond

2020-12-23 19:50

administrator   ~0000062

Hi,

this behaviour is caused by the cloudflare Captcha page. Somewhere in the middle of downloading the story, the captcha page appears and is of course not solved. So no further chapters are downloaded. Not sure if it'll help if I just add a Sleep of maybe 5 sec between chapters, I need to test this.

Cloudflare adds a multiple second delay to check if you are human. that'S why it takes longer if it's working compared to being faster if you are stuck at the error-page

If this won't solve the issue then I'm going for plan B: It'll just stop downloading when a captcha page appears, Show you the page to resolve the captcha and continue after you've confirmed that the chapter is now visible. Simple in theory But the internal structure of FFDL is not prepared for that so it'll take some time. But with this approach FFDL will behave like any human controlled browser.

I hope you are able to survive for now with retrying some downloads, I'm working on the issue

abzb13

2020-12-23 21:51

reporter   ~0000064

I'm happy to hear that the issue is resolvable :)

No worries, worst case I'll just make up a list of the long fics that update in the meantime, and save them for later.

hexarisius

2020-12-24 16:24

reporter   ~0000065

Thanks for the update Raimond. Like abzb13 wrote ill collect the stories and download them later. Keep up the good work :D

Raimond

2020-12-25 14:48

administrator   ~0000066

Update:

I'm currently testing a version which brings up the captcha page for you to manually resolve. Works fine in my tests, sending it currently to an user for testing (Thanks Mike).

I'm also playing with a delay between the individual chapters to reduce the amout of captchas that pop up.

Testing these features is very difficult for me since it seems that my IP is very much trusted, so it nearly never comes up for me

Raimond

2020-12-28 22:19

administrator   ~0000073

Please try 0.9.3, it should fix the problem.
In the settings of FFDL you have two new options to configure delays (random and static one).
Also, capt has are now shown to you for manually solving them if they come up.
Please report if this solves your issue.

abzb13

2020-12-28 22:50

reporter   ~0000074

Awesome!

I'm going through my list of fics that were having problems, from the shortest to the longest, no problems at all!

Issue History

Date Modified Username Field Change
2020-12-22 19:37 abzb13 New Issue
2020-12-22 19:37 abzb13 Status new => assigned
2020-12-22 19:37 abzb13 Assigned To => Raimond
2020-12-22 19:37 abzb13 File Added: Screenshot_20201222-133533_Aldiko Classic Premium.jpg
2020-12-23 15:44 hexarisius Note Added: 0000059
2020-12-23 19:50 Raimond Note Added: 0000062
2020-12-23 21:51 abzb13 Note Added: 0000064
2020-12-24 16:24 hexarisius Note Added: 0000065
2020-12-25 14:48 Raimond Note Added: 0000066
2020-12-28 22:19 Raimond Note Added: 0000073
2020-12-28 22:50 abzb13 Note Added: 0000074