Set MAX_COLUMN to 32000

[ghost]

Hi,

I want to use sqlite3 via Python with a large number of columns. I'm able to compile the amalgation with the right flag set.
I've tested with python3 setup.py build_ext --definevalues SQLITE_MAX_COLUMN=32000 and also
CFLAGS="-DSQLITE_MAX_COLUMN=32000" ./configure and running sqlite3 from the command line shows that the number of columns has been increased (pragma compile_options;).

But when I then build apsw python3 setup.py build & python3 setup.py install and run the shell python3 -m apsw and check the compile options I still see 2000.

What am I missing here? Completely new to apsw but it seamed as a good way to achive what I'm after.

Any help is much appreciated!

[rogerbinns]

Short answer is run this which combines everything into one step:

python3 setup.py fetch --all build_ext --definevalues  SQLITE_MAX_COLUMN=32000 install

You'll note messages about install going away and the above will stop working at some point. This is the proper way of doing it.

Create a setup.apsw file like the following. It is the pypi template with your definevalues added:

[build_ext]
definevalues = SQLITE_MAX_COLUMN=32000

[build]
# download corresponding sqlite release
fetch = True
# all extensions included
enable_all_extensions = True
# for Cursor.description_full
enable = COLUMN_METADATA

Run this to make a wheel:

python3 -m pip wheel .

Then install the wheel whose filename will differ based on your Python version and cpu:

python3 -m pip install apsw-3.42.0.0-cp311-cp311-linux_x86_64.whl

[ghost]

Thx for thte quick answer. It tried both approaches (and I uninstalled apsw before installing) but I still see this in the shell.

sqlite> pragma compile_options;
┌─────────────────────────────────┐
│         compile_options         │
│ ATOMIC_INTRINSICS=1             │
│ COMPILER=clang-12.0.5           │
│ DEFAULT_AUTOVACUUM              │
│ DEFAULT_CACHE_SIZE=-2000        │
│ DEFAULT_FILE_FORMAT=4           │
│ DEFAULT_JOURNAL_SIZE_LIMIT=-1   │
│ DEFAULT_MMAP_SIZE=0             │
│ DEFAULT_PAGE_SIZE=4096          │
│ DEFAULT_PCACHE_INITSZ=20        │
│ DEFAULT_RECURSIVE_TRIGGERS      │
│ DEFAULT_SECTOR_SIZE=4096        │
│ DEFAULT_SYNCHRONOUS=2           │
│ DEFAULT_WAL_AUTOCHECKPOINT=1000 │
│ DEFAULT_WAL_SYNCHRONOUS=2       │
│ DEFAULT_WORKER_THREADS=0        │
│ ENABLE_API_ARMOR                │
│ MALLOC_SOFT_LIMIT=1024          │
│ MAX_ATTACHED=125                │
│ MAX_COLUMN=2000                 │
│ MAX_COMPOUND_SELECT=500         │
│ MAX_DEFAULT_PAGE_SIZE=8192      │
│ MAX_EXPR_DEPTH=1000             │
│ MAX_FUNCTION_ARG=127            │
│ MAX_LENGTH=1000000000           │
│ MAX_LIKE_PATTERN_LENGTH=50000   │
│ MAX_MMAP_SIZE=0x7fff0000        │
│ MAX_PAGE_COUNT=1073741823       │
│ MAX_PAGE_SIZE=65536             │
│ MAX_SQL_LENGTH=1000000000       │
│ MAX_TRIGGER_DEPTH=1000          │
│ MAX_VARIABLE_NUMBER=32766       │
│ MAX_VDBE_OP=250000000           │
│ MAX_WORKER_THREADS=8            │
│ MUTEX_PTHREADS                  │
│ SYSTEM_MALLOC                   │
│ TEMP_STORE=1                    │
│ THREADSAFE=1                    │
└─────────────────────────────────┘

[rogerbinns]

You can use apsw.__file__ to double check where it is installed. Here is a console session showing everything working.

$ python3 -m venv .
$ . bin/activate
(max_column) $ wget -q https://github.com/rogerbinns/apsw/releases/download/3.42.0.0/apsw-3.42.0.0.zip
(max_column) $ unzip -q apsw-3.42.0.0.zip
(max_column) $ cd apsw-3.42.0.0/
(max_column) $ cat > setup.apsw
[build_ext]
definevalues = SQLITE_MAX_COLUMN=32000

[build]
# download corresponding sqlite release
fetch = True
# all extensions included
enable_all_extensions = True
# for Cursor.description_full
enable = COLUMN_METADATA
(max_column) $ pip wheel .
Processing /space/max_column/apsw-3.42.0.0
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: apsw
  Building wheel for apsw (pyproject.toml) ... done
  Created wheel for apsw: filename=apsw-3.42.0.0-cp311-cp311-linux_x86_64.whl size=3941904 sha256=f094c05b6397d79bce2dec141f1287654d715f59e2f5a44b5347ec3938fb8a72
  Stored in directory: /home/rogerb/.cache/pip/wheels/87/c9/68/e13ff80fdee3d4d63ebbf8e6232e0608eab4e5ac37ab9f9ed9
Successfully built apsw
(max_column) $ pip install apsw-3.42.0.0-cp311-cp311-linux_x86_64.whl 
Processing ./apsw-3.42.0.0-cp311-cp311-linux_x86_64.whl
Installing collected packages: apsw
Successfully installed apsw-3.42.0.0
(max_column) $ python3
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import apsw
>>> apsw.__file__
'/space/max_column/lib/python3.11/site-packages/apsw/__init__.cpython-311-x86_64-linux-gnu.so'
>>> [co for co in apsw.compile_options if "COLUMN" in co]
['ENABLE_COLUMN_METADATA', 'MAX_COLUMN=32000']