Sample code for bbot as a module doesn't work in pool or without start_method of fork

[GhostDog98]

Describe the bug
If we run the sample code provided in the advanced usage page:

scan = Scanner(domain, presets=["subdomain-enum"])
for result in scan.start():
        print(result)

In python, we get the correct output.
However, if we are running inside of python, in a Pool (even a pool of 1 item, and you have to use the fork start method or it does not work!), this causes an infinite loop and error whenever you call scan.start()

The fork issue means you need to do:
mp.set_start_method("fork", force=True) at the start of files, if:

• Your file is a module, and is importing another local module
  • That other module executes things with the multiprocessing module as it initialization and doesn't wrap it in a __main__ test case.

Producing this error:

This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Expected behavior
BBot should run without issue under a multiprocessing pool with default settings.

BBOT Command
See above

OS, BBOT Installation Method + Version
OS: Kali linux on WSL
BBOT Version: 2.0.1
Python version: 3.11.8
Install method: pip install bbot==2.0.1

[GhostDog98]

Also I should note that using the fork method is no longer the default as noted in the multiprocessing page:

[GhostDog98]

Update, here's some minimal reproductions:
freeze support issue:

from multiprocessing import Pool
from bbot.scanner import Scanner

def do_things_with_bbot(domain):
    print("Starting scan...")
    scan = Scanner(domain, presets=["subdomain-enum"])
    for result in scan.start():
        print(result)

pool = Pool(1) # Pool of size 1
pool.map(do_things_with_bbot, ["example.com"])

Infinite recursion issue:

from multiprocessing import Pool
import multiprocessing as mp
from bbot.scanner import Scanner
mp.set_start_method("fork", force=True)


def do_things_with_bbot(domain):
    print("Starting scan...")
    scan = Scanner(domain, presets=["subdomain-enum"])
    for result in scan.start():
        print(result)

pool = Pool(1) # Pool of size 1
pool.map(do_things_with_bbot, ["example.com"])

[TheTechromancer]

@GhostDog98 thanks for the detailed report.

This is a known quirk, and exists because BBOT inherently does a heavy amount of parallelization, including spawning its own process pool. Running BBOT in a pool is highly discouraged, as there are better ways to parallelize it, such as turning up the number of threads for the dns engine, modules, etc. Also, since BBOT has global state, running multiple BBOT scans within the same python environment is not recommended.

However, if you really want to do it, you might be able to get it to work by checking the name of the process. This is how BBOT handles it internally:

import multiprocessing

current_process = multiprocessing.current_process()
if current_process.name == "MainProcess":
    # do stuff

[GhostDog98]

Awesome, thank you very much. Unfortunately the use-case for our program requires additional parallelization of bbot together with other tools. I will report back if this works. Another possibility could be running it through docker, and I'll experiment with this too 👍

[TheTechromancer]

We now have a dedicated issue for fixing this bug:

• BBOT python API should work inside non-main process #1904