Pandas Dataframe: How to Add Rows & Columns

Pandas is a popular data manipulation library in Python, widely used for data analysis and data science tasks. Pandas Dataframe is a two-dimensional labeled data structure with columns of potentially different types, similar to a spreadsheet or SQL table. One of the common tasks in data manipulation is adding new rows or columns to an existing dataframe. It might seem like a trivial task, but choosing the right method to add rows or columns can significantly impact the performance and efficiency of your code.

In this blog, we will explore the different ways to add rows and columns to a Pandas Dataframe. We will look into different methods available in Pandas, such as .loc, .iloc, .append, .concat, and many more.  While working on a data project using Python programming, there are several scenarios when you’ll need to add new rows and columns to your Dataframe. In this article, we will show you how to do it. As data scientists or data analysts, you must get a good understanding of how to add Dataframe rows and columns.

In this post, we will work with the following Pandas data frame.

import pandas as pd

df = pd.DataFrame({
    "Mathematics": [95, 99],
    "Science": [98, 94]
}, index=["Aiyana", "Anisha"])

df

How to Add a Row in a Dataframe

There are multiple ways of adding rows to Dataframe. You can use Dataframe.loc or Dataframe.append method to add a row at the end of Dataframe. If you want to insert a row at any specific position, then you can use Dataframe.insert() method. Let’s see all these methods one by one with an example.

Method 1: Add a new row at the End using loc method

In this method, we will take the help of the Dataframe.loc method which is used to access a group of rows and columns by label(s). The loc method of Pandas DataFrame allows users to select subsets of data from a DataFrame based on specific row and column labels. The loc method stands for “location” and is used to filter data by specifying the row and column indices.

Add a new row using .loc[label]: If you want to add a single row to your Dataframe, you can use the .loc[] indexing method. The .loc[] method is used to access Dataframe elements by label, and it supports adding new rows as well as creating copies of existing ones. The following code represents how to add a Dataframe row at the end.

# Use loc method to add a new row with label
#
df.loc["Saanvi"] = [96, 90]

Add a new row using .loc[len(df)]: In the above code, the rows got added using label as index. You could also use loc method to add a new row to dataframe which does not have labels defined as indices. In the code given below, we first created a sample DataFrame with two rows and three columns. We then defined a new row to be added to the DataFrame as a dictionary with the same keys (Name, Mathematics, Science) as the DataFrame columns. We, then, used the .loc method to add the new row to the DataFrame at the index position of the last row (which is obtained using len(df)). Here is the Python code:

import pandas as pd

# Create a sample DataFrame
data = {'Name': ['Aiyana', 'Anisha'],
        'Mathematics': [95, 98],
        'Science': [99, 94]}
df = pd.DataFrame(data)

# Display the original DataFrame
print("Original DataFrame:")
print(df)

# Define the new row to be added
new_row = {'Name': 'Saanvi', 'Mathematics': 96, 'Science': 90}

# Use the loc method to add the new row to the DataFrame
df.loc[len(df)] = new_row

# Display the modified DataFrame
print("\n\nModified DataFrame:")
print(df)

This is what will get printed.

The loc method works by specifying the row index and the column labels where we want to add the new data. In the above two cases, we specified the row index as label and len(df) respectively, which is the next available index after the last row of the DataFrame.

Method 2: Add a new row at the End using append method (deprecated)

In this method, we will take the help of Dataframe.append() method. Dataframe.append() is used to append rows of other Dataframes to the end of this Dataframe, returning a new object. Rows are added at the bottom, so the index labels are increasing, and duplicate index values are not preserved. The code below represents the same:

# Append one or more rows of another dataframe
#
df1 = pd.DataFrame({
    "Mathematics": [92],
    "Science": [95]
}, index=["Snehal"])
#
# Append a dataframe
#
df = df.append(df1)

Update [14 May, 2023]: While executing append method in Google Colab, I got the following alert:

FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. df = df.append(df1)

The append method in Pandas is deprecated and will be removed in a future version. This is because the append method is not as efficient as the concat method, and it can also lead to unexpected results.

The concat method, shown in the next section, is rather recommended method to append DataFrames in Pandas. The concat method has several advantages over the append method. First, the concat method is more efficient. The concat method can append DataFrames in-place, which means that it does not need to create a new copy of the DataFrame. This can be a significant performance improvement for large DataFrames. Second, the concat method is more flexible. The concat method allows you to specify how the rows from the input DataFrames should be aligned. For example, you can specify that the rows should be aligned by index, or by column.

Method 3: Add new rows at the End using concat method

The concat method can be used to add rows to a Pandas Dataframe. The concat method takes an iterable of Series or Dataframe objects and concatenates them into a single Dataframe. The concat method can be used to combine two or more Dataframes into a single Dataframe, or to combine a Series and a Dataframe into a single Dataframe. The following code represents way to add one or more rows to the end of Dataframe.

import pandas as pd
#
# Create a dataframe
#
dict = {"Mathematics":[95, 90, 99],
        "Science": [99, 95, 92]}

df1 = pd.DataFrame(dict, index=["Aiyana", "Anisha", "Saanvi"])
#
# Create another dataframe
#
df2 = pd.DataFrame({"Mathematics": [96],
                    "Science": [99]},
                   index=["Snehal"])
#
# Concat dataframes
#
pd.concat([df1, df2])

In case, you don’t want to add index (such as “Snehal” in the above example), here is the updated code. Make a note of the argument (ignore_index=True) passed to concat method.

import pandas as pd
#
# Create a dataframe
#
dict = {"Mathematics":[95, 90, 99],
        "Science": [99, 95, 92]}

df1 = pd.DataFrame(dict, index=["Aiyana", "Anisha", "Saanvi"])
#
# Create another dataframe
#
df2 = pd.DataFrame({"Mathematics": [96],
                    "Science": [99]})
#
# Concat dataframes
#
pd.concat([df1, df2], ignore_index=True)

When to use which method to add row in dataframe: .loc, .append or .concat

The choice between using .loc, .append, or .concat to add one or more rows to a Pandas dataframe depends on the specific use case and desired outcome.

• Use the .loc method when you want to add a row to a specific index position in the dataframe. This method allows you to specify the index location where you want to add the new row and provide the values for the row as a dictionary.
• Use the .concat method when you want to combine two or more dataframes, either vertically or horizontally. This method can be used to add rows or columns to a dataframe, depending on the axis parameter.
• The .append method has been deprecated and is scheduled to be removed from pandas in a future version. Therefore, for maintaining compatibility with future versions of pandas, it is recommended to use the .concat method

How to Add a column in a Dataframe

There are multiple ways of adding columns to Dataframe. You can use Dataframe.loc or bracket method to add a new column at the end of Dataframe. Let’s see all these methods one by one with an example.

Method 1: Add a new Column at the End using loc method

If you want to add a single column to your Dataframe, you can use the .loc[] indexing method. The .loc[] method is used to access Dataframe elements by label, and it supports adding new columns as well as creating copies of existing ones. The following code represents how to add a Dataframe column at the end.

# Adding a new column using loc method
#
df.loc[:, ["English"]] = [85, 92, 79, 87]

Method 2: Add a new Column at the End using brackets

In this method, we will take the help of using brackets on data frame object to insert a new column. The column is inserted at the end of all the columns. The following code represents how to add columns using brackets.

# Adding a new column using brackets
#
df["Hindi"] = [81, 79, 72, 76]

Method 3: Add a new Column at the End using insert method

In this method, we will take the help of Dataframe.insert() method. Dataframe.insert() is used to insert a column in Dataframe at a specified location. The column is inserted at the given position among all columns. The following code represents how to add a Dataframe column using Dataframe.insert() method:

# Adding a column at a specified position using insert method
#
df.insert(1, "Social Science", [86, 78, 82, 80])

Method 4: Add a new Column using Concat Method

The following code demonstrates that you can also use concat method with axis=”columns” to add a new column to existing data frame. Here is the code:

import pandas as pd

#
# Create a new dataframe
#
data = {'Name': ['John', 'Alice', 'Bob'],
        'Age': [25, 30, 35]}
df = pd.DataFrame(data)
#
# Define a new column dataframe
#
new_column_df = pd.DataFrame({'Salary': [5000, 6000, 7000]})
#
# Add new column using concat method
#
df = pd.concat([df, new_column_df], axis="columns")

df

When to use which method to add columns: .loc, .insert, label, .concat

The choice between using .loc, .insert, or adding at the end of columns to add one or more columns to a Pandas dataframe depends on the specific use case and desired outcome. Here are some guidelines to help you decide which method to use:

• Use the .loc method when you want to add a new column with specific values at a specific index position in the dataframe. This method allows you to specify the index location where you want to add the new column and provide the values for the column as a list or an array.
• Use the .insert method when you want to add a new column at a specific index position and shift the existing columns to the right. This method allows you to specify the index location where you want to add the new column, provide a name for the column, and provide the values for the column as a list or an array.
• Use the “label” (adding at the end of columns) method when you want to add a new column at the end of the dataframe. This method allows you to append a new column to the existing dataframe.
• Use the .concat method when you have a separate DataFrame or Series with the values for the new column. It allows you to concatenate DataFrames horizontally, aligning the data based on the index or column labels. This method is useful when you want to combine multiple DataFrames or Series along a common axis.

Conclusion

In this blog, we looked at various methods to manipulate Pandas DataFrames, specifically focusing on adding rows and columns. Three potent techniques for adding rows were discussed — namely, the ‘loc‘, ‘append‘, and ‘concat‘ methods. Similarly, three efficient approaches for adding columns got discussed, which include the ‘loc‘, ‘insert‘, and ‘brackets‘ methods. When incorporating new rows, they must align with the existing structure, having the same number of columns. Likewise, any new columns should maintain consistency in data type with the existing ones. For large-scale DataFrame modifications, consider using the ‘inplace‘ parameter to avoid producing a new DataFrame copy. I hope this blog post has been insightful and has equipped you with practical tools for your data manipulation tasks related to adding rows and columns to Pandas Dataframe. However, please feel free to reach out in case you need further clarifications. Feel free to suggest if I missed on any points.