R Data Frame – Remove Duplicate Rows

Remove Duplicate Rows from an R Data Frame

Duplicate rows can occur when data is imported, combined, or recorded more than once. In R, you can remove complete duplicate rows with unique() or duplicated(). You can also remove duplicates based on selected columns while deciding which row to retain.

Remove Complete Duplicate Rows with unique()

To remove rows whose values are duplicated across every column of an R data frame, pass the data frame to unique().

 newDataFrame = unique(redundantDataFrame)

In this syntax:

• redundantDataFrame is the data frame containing one or more duplicate rows.
• newDataFrame receives the rows that remain after duplicates are removed.
• unique() compares complete rows and retains the first occurrence of each distinct row.

Example: Remove Identical Rows from an R Data Frame

In this example, the first and fourth rows contain the same values in all three columns. The unique() function removes the later occurrence.

> DF1 = data.frame(C1= c(1, 5, 14, 1, 54), C2= c(9, 15, 85, 9, 42), C3= c(8, 7, 42, 8, 16))
> DF1
  C1 C2 C3
1  1  9  8
2  5 15  7
3 14 85 42
4  1  9  8
5 54 42 16
>

Row 1 and Row 4 are duplicates. When we run unique() function, it retains the first row which is original and any duplicates further in the data frame are removed.

> DF2 = unique(DF1)
> DF2
  C1 C2 C3
1  1  9  8
2  5 15  7
3 14 85 42
5 54 42 16
>

The original row names are retained, so the result contains row names 1, 2, 3, and 5. Reset them when consecutive row names are required.

DF2 <- unique(DF1)
rownames(DF2) <- NULL
DF2

  C1 C2 C3
1  1  9  8
2  5 15  7
3 14 85 42
4 54 42 16

Find Duplicate Rows Before Removing Them

Use duplicated() when you need to inspect which rows are repeated. It returns a logical vector in which TRUE marks a row that duplicates an earlier row.

duplicated(DF1)

[1] FALSE FALSE FALSE TRUE FALSE

To display only the later duplicate occurrences, use the logical vector for row filtering.

DF1[duplicated(DF1), ]

  C1 C2 C3
4  1  9  8

To display every row that belongs to a duplicated group, including the first occurrence, test duplicates in both directions.

duplicate_group <- duplicated(DF1) | duplicated(DF1, fromLast = TRUE)
DF1[duplicate_group, ]

  C1 C2 C3
1  1  9  8
4  1  9  8

Remove Duplicate Rows with duplicated()

You can remove complete duplicate rows by retaining rows for which duplicated() returns FALSE.

newDataFrame <- redundantDataFrame[!duplicated(redundantDataFrame), ]

The exclamation mark negates the logical values. Therefore, later duplicates marked TRUE are excluded.

DF3 <- DF1[!duplicated(DF1), ]
DF3

  C1 C2 C3
1  1  9  8
2  5 15  7
3 14 85 42
5 54 42 16

Remove Duplicate Rows Based on One Column

Sometimes rows are considered duplicates when a key column repeats, even though other column values differ. Apply duplicated() to that column and use the result to filter the complete data frame.

employees <- data.frame(
  employee_id = c(101, 102, 101, 103),
  name = c("Anita", "Bala", "Anita Rao", "Charan"),
  score = c(82, 75, 91, 88)
)

employees_unique <- employees[!duplicated(employees$employee_id), ]
employees_unique

  employee_id   name score
1         101  Anita    82
2         102   Bala    75
4         103 Charan    88

This keeps the first row for each employee_id. Values in the other columns are not used to determine whether the row is duplicated.

Remove Duplicate Rows Based on Multiple Columns

To identify duplicates using a combination of columns, pass those columns as a smaller data frame to duplicated().

orders <- data.frame(
  customer_id = c(1, 1, 1, 2, 2),
  order_date = as.Date(c("2026-07-01", "2026-07-01", "2026-07-02", "2026-07-01", "2026-07-01")),
  amount = c(120, 150, 90, 200, 200)
)

key_columns <- orders[c("customer_id", "order_date")]
orders_unique <- orders[!duplicated(key_columns), ]
orders_unique

  customer_id order_date amount
1           1 2026-07-01    120
3           1 2026-07-02     90
4           2 2026-07-01    200

Rows are treated as duplicates when both customer_id and order_date match. The amount column does not participate in the comparison.

Keep the Last Duplicate Row Instead of the First

By default, duplicated() marks later occurrences as duplicates. Set fromLast = TRUE to mark earlier occurrences instead, which lets you retain the last row in each duplicated group.

employees_last <- employees[!duplicated(employees$employee_id, fromLast = TRUE), ]
employees_last

  employee_id      name score
2         102      Bala    75
3         101 Anita Rao    91
4         103    Charan    88

For employee_id 101, the later row containing the score 91 is retained.

Remove Duplicates Based on a Condition

When the row to keep depends on a value such as the newest date, highest score, or preferred status, sort the data first and then remove duplicates by the key column.

records <- data.frame(
  id = c(1, 1, 2, 2),
  updated_on = as.Date(c("2026-06-10", "2026-07-20", "2026-07-01", "2026-06-15")),
  status = c("pending", "approved", "active", "inactive")
)

records_sorted <- records[order(records$id, records$updated_on, decreasing = TRUE), ]
latest_records <- records_sorted[!duplicated(records_sorted$id), ]
latest_records <- latest_records[order(latest_records$id), ]
rownames(latest_records) <- NULL
latest_records

  id updated_on   status
1  1 2026-07-20 approved
2  2 2026-07-01   active

The data is ordered so that the newest record for each id appears first. Removing duplicates then retains that newest record.

Remove Duplicate Rows with dplyr distinct()

In a dplyr workflow, use distinct() to remove duplicates. With no column names, it compares complete rows.

library(dplyr)

DF1 %>%
  distinct()

To remove duplicates based on one or more selected columns while retaining every column, list the key columns and set .keep_all = TRUE.

employees %>%
  distinct(employee_id, .keep_all = TRUE)

For multiple-column matching, include each key column.

orders %>%
  distinct(customer_id, order_date, .keep_all = TRUE)

Like the base R methods shown above, distinct() keeps the first matching row unless you arrange the data in the required priority order beforehand.

Count Duplicate Rows in an R Data Frame

Use sum(duplicated(dataFrame)) to count later duplicate occurrences.

sum(duplicated(DF1))

[1] 1

To count duplicates based on selected columns, apply duplicated() to those columns.

sum(duplicated(orders[c("customer_id", "order_date")]))

Remove Duplicate Columns from an R Data Frame

Duplicate columns require a different comparison because duplicated(dataFrame) examines rows. Transpose the data frame before applying duplicated(), and use the resulting logical vector to select columns.

data_with_duplicate_columns <- data.frame(
  A = c(1, 2, 3),
  B = c(4, 5, 6),
  C = c(1, 2, 3)
)

without_duplicate_columns <- data_with_duplicate_columns[
  !duplicated(as.list(data_with_duplicate_columns))
]
without_duplicate_columns

  A B
1 1 4
2 2 5
3 3 6

Choosing the Correct R Duplicate-Removal Method

Requirement	Recommended approach
Remove completely identical rows	unique(dataFrame)
Inspect duplicate row positions	duplicated(dataFrame)
Remove duplicates using one column	dataFrame[!duplicated(dataFrame$key), ]
Remove duplicates using multiple columns	Apply duplicated() to a subset of columns
Keep the last occurrence	Use fromLast = TRUE
Keep a row based on date, score, or status	Sort by priority before removing duplicates
Remove duplicates in a dplyr pipeline	Use distinct()

Common Problems When Removing Duplicate Rows in R

• Comparing every column unintentionally: unique() removes a row only when all column values match. Select key columns when only certain fields define a duplicate.
• Keeping the wrong occurrence: Base R and dplyr::distinct() normally retain the first occurrence. Sort the data first when the last, newest, or highest-priority row should remain.
• Ignoring differences in text: Values such as "R Tutorial", "r tutorial", and "R Tutorial " are different. Standardize capitalization and whitespace before deduplication when appropriate.
• Treating missing values incorrectly: Check how NA values appear in the key columns and confirm that rows with missing keys should be grouped together.
• Overwriting the source too early: Assign the result to a new object until the retained rows have been verified.

Frequently Asked Questions About Duplicate Rows in R

Does unique() keep the first or last duplicate row in R?

unique() keeps the first occurrence of each complete row. To keep the last occurrence based on a key column, use !duplicated(key, fromLast = TRUE).

How do I remove duplicates in R based on one column?

Filter the complete data frame with !duplicated(dataFrame$columnName). This retains the first row for each distinct value in that column.

How do I remove duplicates based on multiple columns in R?

Create a subset containing the key columns and pass it to duplicated(). In dplyr, list the columns in distinct() and use .keep_all = TRUE to retain the other columns.

How can I find duplicates without deleting them?

Use duplicated(dataFrame) to obtain a logical vector. Filter with dataFrame[duplicated(dataFrame), ] to display later duplicate occurrences.

How do I keep the newest row for each ID?

Sort the data by ID and date so the newest row appears first within each ID. Then remove duplicates based on the ID column. The first retained row for each ID will be the newest one.

R Duplicate-Row Tutorial QA Checklist

• Confirm whether duplicates should be identified across complete rows, one key column, or several key columns.
• Verify whether the first, last, newest, or highest-priority matching row must be retained.
• Check whether text values require trimming or case normalization before comparison.
• Review how missing values in duplicate-key columns should be handled.
• Compare row counts before and after deduplication and inspect a sample of removed records.

Summary of Removing Duplicate Rows in R

Use unique() for a direct removal of completely identical rows. Use duplicated() when you need to inspect duplicates, select the comparison columns, or control whether the first or last occurrence remains. In a dplyr workflow, distinct() provides equivalent operations with concise column selection.

In this R Tutorial, we have learnt how to remove duplicate rows in R Data frame.