DOC: `.str.cat` output in case of `Index` object #35556

galipremsagar · 2020-08-04T21:34:20Z

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
(optional) I have confirmed this bug exists on the master branch of pandas.

Code Sample, a copy-pastable example

import pandas as pd
import numpy as np
i = pd.Index(["AbC", "de", "FGHI", "j", "kLm"])
other = (
            pd.Series(["f", "g", "h", "i", "j"]),
            np.array(["f", "a", "b", "f", "a"]),
            pd.Series(["f", "g", "h", "i", "j"]),
            np.array(["f", "a", "b", "f", "a"]),
            np.array(["f", "a", "b", "f", "a"]),
            pd.Index(["1", "2", "3", "4", "5"]),
            np.array(["f", "a", "b", "f", "a"]),
            pd.Index(["f", "g", "h", "i", "j"]),
)
print(i.str.cat(other))
x = pd.Series(["f", "g", "h", "i", "j"])
print(i.str.cat(x))

0.25.3 output:

Index(['AbCfffff1ff', 'degagaa2ag', 'FGHIhbhbb3bh', 'jififf4fi',
       'kLmjajaa5aj'],
      dtype='object')
Index(['AbCf', 'deg', 'FGHIh', 'ji', 'kLmj'], dtype='object')

1.1.0 output:

Index([nan, nan, nan, nan, nan], dtype='object')
Index([nan, nan, nan, nan, nan], dtype='object')

Problem description

The output of 0.25.3 version seems to be correct as when we change i to be a Series the incorrect results go away:

import pandas as pd
import numpy as np
i = pd.Series(["AbC", "de", "FGHI", "j", "kLm"])
other = (
            pd.Series(["f", "g", "h", "i", "j"]),
            np.array(["f", "a", "b", "f", "a"]),
            pd.Series(["f", "g", "h", "i", "j"]),
            np.array(["f", "a", "b", "f", "a"]),
            np.array(["f", "a", "b", "f", "a"]),
            pd.Index(["1", "2", "3", "4", "5"]),
            np.array(["f", "a", "b", "f", "a"]),
            pd.Index(["f", "g", "h", "i", "j"]),
)
print(i.str.cat(other))
x = pd.Series(["f", "g", "h", "i", "j"])
print(i.str.cat(x))


0     AbCfffff1ff
1      degagaa2ag
2    FGHIhbhbb3bh
3       jififf4fi
4     kLmjajaa5aj
dtype: object
0     AbCf
1      deg
2    FGHIh
3       ji
4     kLmj
dtype: object

Expected Output

The output should be similar to Series behaviour/ 0.25.3 behaviour.

Output of `pd.show_versions()`

INSTALLED VERSIONS

commit : d9fff27
python : 3.7.3.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Sun Jul 5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8
pandas : 1.1.0
numpy : 1.19.0
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 49.1.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

dsaxton · 2020-08-05T01:13:35Z

I think this is expected as per the warning from 0.25.3 (that the operation is first going to perform index alignment in a future version):

In [1]: import pandas as pd                                                                                                                                                                                  
In [2]: idx = pd.Index(["a", "b", "c", "d", "e"])                                                                                                                                                            
In [3]: ser = pd.Series(["f", "g", "h", "i", "j"])                                                                                                                                                           
In [4]: print(idx.str.cat(ser))                                                                                                                                                                              
<ipython-input-4-157619096d44>:1: FutureWarning: A future version of pandas will perform index alignment when `others` is a Series/Index/DataFrame (or a list-like containing one). To disable alignment (the behavior before v.0.23) and silence this warning, use `.values` on any Series/Index/DataFrame in `others`. To enable alignment and silence this warning, pass `join='left'|'outer'|'inner'|'right'`. The future default will be `join='left'`.
  print(idx.str.cat(ser))
/Users/danielsaxton/opt/miniconda3/envs/pandas-0.25.3/lib/python3.8/site-packages/numpy/core/fromnumeric.py:87: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
Index(['af', 'bg', 'ch', 'di', 'ej'], dtype='object')

In [5]: print(pd.__version__)                                                                                                                                                                                
0.25.3

Following the suggestion from the warning and casting to numpy first we get the desired output:

In [1]: import pandas as pd                                                                                                                                                                                  
In [2]: idx = pd.Index(["a", "b", "c", "d", "e"])                                                                                                                                                            
In [3]: ser = pd.Series(["f", "g", "h", "i", "j"])                                                                                                                                                           
In [4]: print(idx.str.cat(ser))                                                                                                                                                                              
Index([nan, nan, nan, nan, nan], dtype='object')

In [5]: print(pd.__version__)                                                                                                                                                                                
1.2.0.dev0+29.ga4203cf8d

In [6]: idx.str.cat(ser.to_numpy())                                                                                                                                                                          
Out[6]: Index(['af', 'bg', 'ch', 'di', 'ej'], dtype='object')

After taking a look I'm not sure if this is explicit enough in the documentation.

simonjayhawkins · 2020-08-05T11:10:11Z

Thanks @galipremsagar for the report and thanks @dsaxton for the answer.

After taking a look I'm not sure if this is explicit enough in the documentation.

would take documentation improvements so will leave this issue open

souris-dev · 2020-08-07T11:46:49Z

Hello, I would like to work on this issue.

simonjayhawkins · 2020-08-07T12:26:32Z

Thanks @souris-dev. If you need any help, just ask.

souris-dev · 2020-08-07T13:11:03Z

take


        DOC: Updated docstring for Series.str.cat, issue pandas-dev#35556


        DOC: Updated docstring for Series.str.cat (pandas-dev#35556)


        DOC: Updated docstring for Series.str.cat, issue pandas-dev#35556


        DOC: Updated docstring for Series.str.cat (pandas-dev#35556)


        DOC: Updated docstring for Series.str.cat (pandas-dev#35556)


        DOC: Updated docstring for Series.str.cat (pandas-dev#35556)


        DOC: Updated docstring for Series.str.cat (pandas-dev#35556)

galipremsagar added Bug Needs Triage labels Aug 4, 2020

galipremsagar changed the title ~~BUG: `.str.cat` results in incorrect output Index object~~ BUG: `.str.cat` results in incorrect output incase of `Index` object Aug 4, 2020

galipremsagar mentioned this issue Aug 4, 2020

[REVIEW] Fix `.str.cat` when objects with different index are passed rapidsai/cudf#5844

Merged

dsaxton added Docs Usage Question and removed Bug Needs Triage labels Aug 5, 2020

simonjayhawkins changed the title ~~BUG: `.str.cat` results in incorrect output incase of `Index` object~~ DOC: `.str.cat` output in case of `Index` object Aug 5, 2020

simonjayhawkins added good first issue and removed Usage Question labels Aug 5, 2020

simonjayhawkins added this to the Contributions Welcome milestone Aug 5, 2020

github-actions bot assigned souris-dev Aug 7, 2020

souris-dev added a commit to souris-dev/pandas that referenced this issue Aug 18, 2020

DOC: Updated docstring for Series.str.cat, issue pandas-dev#35556

f88843d

souris-dev added a commit to souris-dev/pandas that referenced this issue Aug 18, 2020

DOC: Updated docstring for Series.str.cat (pandas-dev#35556)

56db6a9

souris-dev added a commit to souris-dev/pandas that referenced this issue Aug 18, 2020

DOC: Updated docstring for Series.str.cat, issue pandas-dev#35556

cacf136

souris-dev added a commit to souris-dev/pandas that referenced this issue Aug 18, 2020

DOC: Updated docstring for Series.str.cat (pandas-dev#35556)

8936c4d

souris-dev added a commit to souris-dev/pandas that referenced this issue Aug 18, 2020

DOC: Updated docstring for Series.str.cat (pandas-dev#35556)

2776514

souris-dev added a commit to souris-dev/pandas that referenced this issue Aug 18, 2020

DOC: Updated docstring for Series.str.cat (pandas-dev#35556)

c3dbf4d

souris-dev added a commit to souris-dev/pandas that referenced this issue Aug 18, 2020

DOC: Updated docstring for Series.str.cat (pandas-dev#35556)

542a42b

souris-dev linked a pull request that will close this issue Aug 18, 2020

DOC: Updated docstring for Series.str.cat (pandas-dev#35556) #35784

Open

4 of 5 tasks complete

Jul	AUG	Sep
	21
2019	2020	2021

pandas-dev / pandas

DOC: `.str.cat` output in case of `Index` object #35556

DOC: `.str.cat` output in case of `Index` object #35556

galipremsagar commented Aug 4, 2020

INSTALLED VERSIONS

dsaxton commented Aug 5, 2020 •

edited

simonjayhawkins commented Aug 5, 2020

souris-dev commented Aug 7, 2020

simonjayhawkins commented Aug 7, 2020

souris-dev commented Aug 7, 2020

pandas-dev / pandas

Sponsor pandas-dev/pandas

Join GitHub today

DOC: `.str.cat` output in case of `Index` object #35556

DOC: `.str.cat` output in case of `Index` object #35556

Comments

galipremsagar commented Aug 4, 2020

Code Sample, a copy-pastable example

Problem description

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

dsaxton commented Aug 5, 2020 • edited

simonjayhawkins commented Aug 5, 2020

souris-dev commented Aug 7, 2020

simonjayhawkins commented Aug 7, 2020

souris-dev commented Aug 7, 2020

Output of `pd.show_versions()`

dsaxton commented Aug 5, 2020 •

edited