Context restoring

README.md

@@ -88,9 +88,9 @@ Here is the list of available actions:
 - `Click(selector, click_options, wait_options)` - click on element on page
 - `Scroll(selector, wait_options)` - scroll page
 - `Screenshot(options)` - take screenshot
+- `RecaptchaSolver(solve_recaptcha, close_on_empty)` - find or solve recaptcha on page
 - `Har()` - to get the HAR file, pass the `har_recording=True` argument to `PuppeteerRequest` at the start of execution.
 - `FillForm(input_mapping, submit_button)` - to fill out and submit forms on page.
-- `RecaptchaSolver(solve_recaptcha)` - find or solve recaptcha on page
 - `CustomJsAction(js_function)` - evaluate JS function on page
 
 Available options essentially mirror [service](https://github.com/ispras/scrapy-puppeteer-service) method parameters, which in turn mirror puppeteer API functions to some extent.
@@ -174,6 +174,42 @@ and will notify you about number of found captchas on the page.
 If you don't want the middleware to work on specific request you may provide special meta key: `'dont_recaptcha': True`.
 In this case RecaptchaMiddleware will just skip the request.
 
+## Automatic context restoring
+
+Sometimes you may receive responses with status 422 (Unprocessable Entity).
+This means the scrapy-puppeteer-services struggled to find provided context or page in its memory.
+In such situations you can use this middleware to restore such contexts.
+
+Enabling the middleware:
+```Python
+DOWNLOADER_MIDDLEWARES = {  # Strict order of middlewares
+    # 'scrapypuppeteer.middleware.PuppeteerRecaptchaDownloaderMiddleware': 1040,  # You may also use recaptcha middleware
+    'scrapypuppeteer.middleware.PuppeteerContextRestoreDownloaderMiddleware': 1041,
+    'scrapypuppeteer.middleware.PuppeteerServiceDownloaderMiddleware': 1042,
+}
+```
+
+Settings of the middleware:
+```Python
+N_RETRY_RESTORING = 3  # Number of tries to restore a context
+RESTORING_LENGTH = 2  # Number of restorable requests in a sequence
+```
+
+Currently, the middleware can only restart from the beginning of request-response sequence.
+You can start this sequence with `recover_context` meta-key, just provide `True` value.
+Example:
+```Python
+...
+yield PuppeteerRequest(
+    url,
+    callback=self.click_on_navigation,
+    errback=self.errback,
+    close_page=False,
+    meta={'recover_context': True}
+)
+...
+```
+
 ## TODO
 
 - [x] skeleton that could handle goto, click, scroll, and actions

examples/spiders/dead_context.py

@@ -0,0 +1,74 @@
+import scrapy
+
+from asyncio import sleep
+
+from scrapypuppeteer import PuppeteerRequest, PuppeteerResponse
+from scrapypuppeteer.actions import Click, GoTo
+from twisted.python.failure import Failure
+
+
+class DeadContextSpider(scrapy.Spider):
+    custom_settings = {
+        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
+        "DOWNLOADER_MIDDLEWARES": {
+            "scrapypuppeteer.middleware.PuppeteerContextRestoreDownloaderMiddleware": 1041,
+            "scrapypuppeteer.middleware.PuppeteerServiceDownloaderMiddleware": 1042,
+        },
+        "N_RETRY_RESTORING": 3,
+        "RESTORING_LENGTH": 2,
+    }
+    name = "dead_context"
+
+    def start_requests(self):
+        urls = [
+            "https://www.google.com/recaptcha/api2/demo",
+            "https://scrapy.org",
+            "https://pptr.dev",
+        ]
+
+        for url in urls:
+            yield PuppeteerRequest(
+                url,
+                callback=self.click_on_navigation,
+                errback=self.errback,
+                close_page=False,
+                meta={"recover_context": True},
+            )
+
+    async def click_on_navigation(self, response: PuppeteerResponse):
+        await sleep(4)
+
+        click = Click(
+            "#__docusaurus > nav > div.navbar__inner > div:nth-child(1) > a:nth-child(3)"
+        )
+        yield response.follow(
+            click, callback=self.click_back, errback=self.errback, close_page=False
+        )
+
+    async def click_back(self, response: PuppeteerResponse):
+        await sleep(4)
+
+        click = Click(
+            "#__docusaurus > nav > div.navbar__inner > div:nth-child(1) > a.navbar__brand > b"
+        )
+        yield response.follow(
+            click, callback=self.goto_api, errback=self.errback, close_page=False
+        )
+
+    async def goto_api(self, response):
+        await sleep(4)
+
+        yield response.follow(
+            GoTo("api/puppeteer.puppeteernode"),
+            callback=self.empty_action,
+            errback=self.errback,
+            close_page=False,
+        )
+
+    @staticmethod
+    async def empty_action(response, **kwargs):
+        await sleep(4)
+
+    @staticmethod
+    def errback(failure: Failure):
+        print(failure)