Error when using --withoutjavascript and a hash in url

[artemf]

I have a problem when using the urls with a hash (as an angular route) and --withoutjavascript command line param. In this scenario minimalcss always fails with Cannot read property 'ok' of null error.

My app might be too complicated for a quick demo, but here's the simplified case that demonstrates the problem:

$ minimalcss --output out.css "https://www.google.com/"
[Success]
$ minimalcss --output out.css "https://www.google.com/#"
[Success]
$ minimalcss --withoutjavascript --output out.css "https://www.google.com/"
[Success]
$ minimalcss --withoutjavascript --output out.css "https://www.google.com/#"
TypeError: Cannot read property 'ok' of null
    at isOk (/usr/local/lib/node_modules/minimalcss/src/run.js:13:35)
    at Promise (/usr/local/lib/node_modules/minimalcss/src/run.js:313:12)
    at process._tickCallback (internal/process/next_tick.js:68:7)

It's not just google:

$ minimalcss --withoutjavascript --output out.css "https://www.bing.com/#"
TypeError: Cannot read property 'ok' of null
    at isOk (/usr/local/lib/node_modules/minimalcss/src/run.js:13:35)
    at Promise (/usr/local/lib/node_modules/minimalcss/src/run.js:313:12)
    at process._tickCallback (internal/process/next_tick.js:68:7)
$ minimalcss --withoutjavascript --output out.css "https://www.yahoo.com/#"
TypeError: Cannot read property 'ok' of null
    at isOk (/usr/local/lib/node_modules/minimalcss/src/run.js:13:35)
    at Promise (/usr/local/lib/node_modules/minimalcss/src/run.js:313:12)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Seems like the no-js run with a hash in url fails.

Would appreciate any help / guidance of what might be happening and why it's failing!

[peterbe]

Note-to-self; The --withoutjavascript is confusing as hell. Here's the help usage:

▶ node bin/minimalcss.js --help | grep without
  --withoutjavascript           The CSS is evaluated against the DOM twice, first with no JavaScript, then with. This disables the load without JavaScript.

Essentially, using --withoutjavascript means it'll open the page twice. That's NOT the default. Contrary to what the help usage description says.

If you omit --withoutjavascript it will just open the page once. Like a regular browser.

Either way, I put some print logs about which URLs it's sending to page.goto():

▶ node bin/minimalcss.js --withoutjavascript --output out.css "https://www.peterbe.com/#" -d
PAGEURL 1 https://www.peterbe.com/#
PAGEURL 2 https://www.peterbe.com/#
response: null
TypeError: Cannot read property 'ok' of null
    at isOk (/Users/peterbe/dev/JAVASCRIPT/minimalcss/src/run.js:13:35)
    at Promise (/Users/peterbe/dev/JAVASCRIPT/minimalcss/src/run.js:316:12)
    at process.internalTickCallback (internal/process/next_tick.js:77:7)

Here's it's having a problem with https://www.peterbe.com/# and the result of await page.goto('https://www.peterbe.com/#', { waitUntil: 'networkidle0' }); is null!

But if you omit, it's the exact same line (await page.goto('https://www.peterbe.com/#', { waitUntil: 'networkidle0' });) but this time it does not return just null.

I wonder, does page.goto() return null when the page is fully cached or something?

[peterbe]

Interesting comment here: puppeteer/puppeteer#2479 (comment)

[peterbe]

Even more interesting: puppeteer/puppeteer#2479 (comment)

What @aslushnikov is suggesting is that the second page goto should be a new fresh page. What minimalcss does is something like this:

// NOTE for this pseudo code demonstration I stripped out the try{}catch{} stuff

urls.forEach(url) {
  const page = await browser.newPage();
  await processPage({
    page,
    options,  // <-- in here it has instructions to tell processPage()
    url,      //     to do the page.goto() 2 times.
    ...
  });
}

What I think needs to happen is something like this:

urls.forEach(url) {
  const perUrlOptions = Object.assign({}, options)
  let page 
  if (perUrloptions.withoutjavascript) {
    page = await browser.newPage();
    await processPage({
      page,
      perUrloptions, 
      url,      
      ...
    });
    delete perUrloptions.withoutjavascript;
  }
  
  page = await browser.newPage();
  await processPage({
    page,
    perUrloptions, 
    url,      
    ...
  });
}

@artemf Would you mind attempting a patch like that and see if it works?

[artemf]

Thanks for the research @peterbe! I will try the suggested fix soon.

[peterbe]

@artemf ping! Had a chance to look at a PR?

[AdamBCo]

Any update on this issue?

[peterbe]

Any update on this issue?

Sadly no. Not me. Is this error happening to you? If so, do you think you can attempt a rough version of the above mentioned potential solution and see if it works?