PERF: selection inside a row using a list of labels 30 slower than selection by the same individual labels

[xstreck1]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this issue exists on the latest version of pandas.

• I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

The code below accesses two columns either together or separately:

import pandas as pd
import numpy as np
import timeit

# Create large DataFrame
n_rows = 100_000
df = pd.DataFrame({
    'a': np.random.random(n_rows),
    'b': np.random.random(n_rows)
})

# Method 1: Access columns inside function
def sum_inside(row):
    return row[0] + row[1]

# Method 2: Pass values directly
def sum_outside(a, b):
    return a + b

# Time Method 1
t1 = timeit.timeit(lambda: df.apply(lambda row: sum_inside(row[['a','b']].values), axis=1), number=1)

# Time Method 2  
t2 = timeit.timeit(lambda: df.apply(lambda row: sum_outside(row['a'], row['b']), axis=1), number=1)

print(f"Method 1 (access inside): {t1:.4f} seconds")
print(f"Method 2 (pass values): {t2:.4f} seconds")

Output:

Method 1 (access inside): 15.6217 seconds
Method 2 (pass values): 0.5135 seconds

Using iloc does not suffer the same issue. Replace:

t1 = timeit.timeit(lambda: df.apply(lambda row: sum_inside(row.iloc[0:2].values), axis=1), number=1)
t2 = timeit.timeit(lambda: df.apply(lambda row: sum_outside(row.iloc[0], row.iloc[1]), axis=1), number=1)

Output:

Method 1 (access inside): 1.5293 seconds
Method 2 (pass values): 0.8811 seconds

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.11.7
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.26100
machine : AMD64
processor : AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_Germany.1252

pandas : 2.2.3
numpy : 1.26.3
pytz : 2024.1
dateutil : 2.8.2
pip : 24.0
Cython : None
sphinx : 8.1.3
IPython : 8.22.1
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.2.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.2
lxml.etree : 4.9.4
matplotlib : 3.8.3
numba : 0.59.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 14.0.2
pyreadstat : None
pytest : 8.2.0
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.12.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

Prior Performance

No response

[rhshadrach]

Thanks for the report. This can be reproduced on just a Series.

ser = pd.Series({"a": 1, "b": 2})

%timeit row[["a", "b"]]
# 103 μs ± 273 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

%timeit row["a"], row["b"]
# 1.76 μs ± 12.4 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

I haven't checked, but likely looking up a single label has a fastpath, and requires less validation. Certainly any perf improvement is always welcome, but I'm not sure there is an issue here.

I presume the example in the OP is just a simplified version of the operation you're trying to perform. In any case, avoiding apply at all costs is the best way to get good performance out of pandas.