Building explainable duplicate detection in Laravel (no ML, no packages)

Question

I built a rental listing platform with Laravel 13, PHP 8.4, PostgreSQL, Redis, and Sanctum.

One problem quickly became obvious:

Agents could post the same property multiple times.

In Nigeria, property listings often don't have precise street addresses. A typical listing might look like:

"2-bedroom flat, Yaba. Near Yaba Tech."

There may be no house number and sometimes no street name.

Instead, people identify properties using a combination of:

• Area
• Landmark
• Property type
• Number of bedrooms
• Rent

That makes duplicate detection a little different from what you might build for a marketplace with structured addresses.

I wanted a solution that was:

• Explainable
• Easy to tune
• Easy for reviewers to understand
• Independent of machine learning
• Cheap enough to run whenever a listing changes

So I built a rule-based duplicate detection system using Laravel and PHP.

---

The Architecture

The basic workflow is:

Property created/updated
        ↓
Find candidate properties
        ↓
Calculate similarity score
        ↓
Store score + explanation
        ↓
Classify match
        ↓
Reviewer confirms/rejects

The important distinction is that I don't compare every property against every other property.

Instead, I first narrow the candidate set using database queries and then perform the more expensive comparison in PHP.

---

1. Creating a Property Fingerprint

Because Nigerian rental listings don't always have reliable addresses, I needed another way to represent a property's identity.

I started with:

Area + Landmark + Bedrooms + Property Type

These fields are normalized and hashed into a fingerprint.

class PropertyFingerprintService
{
    public static function generate(array $data): string
    {
        $parts = [
            strtolower(trim($data['area'] ?? '')),
            strtolower(trim($data['landmark'] ?? '')),
            (string) ($data['bedrooms'] ?? ''),
            strtolower(trim($data['property_type'] ?? '')),
        ];

        return hash('sha256', implode('|', $parts));
    }
}

For example:

Yaba | Near Yaba Tech | 2 | Flat

becomes a deterministic SHA-256 hash.

This gives us a fast way to identify listings that have the same core identity fields.

---

An Important Database Lesson

Initially, I made the fingerprint column unique:

$table->string('fingerprint')->unique();

That turned out to be a mistake.

If an agent tried to create the same property again, the database rejected it.

That sounds useful until you remember the purpose of the feature.

I don't want the database to prevent the duplicate.

I want the duplicate to exist so that the platform can surface it for human review.

So I changed the migration to:

$table->dropUnique(['fingerprint']);
$table->index('fingerprint');

Now multiple properties can have the same fingerprint.

The database helps us find duplicates instead of blocking them.

That distinction is important for any system where duplicates are supposed to be reviewed rather than automatically rejected.

---

2. Candidate Selection

The next problem is scale.

Imagine the platform has 100,000 properties.

Comparing one new property against all 100,000 would obviously be wasteful.

Instead, I narrow the candidates using fields that are already indexed.

The query looks roughly like this:

public function findCandidates(Property $property): EloquentCollection
{
    return Property::query()
        ->where('id', '!=', $property->id)
        ->where(function ($query) use ($property) {
            $query->where('fingerprint', $property->fingerprint)
                ->orWhere(function ($inner) use ($property) {
                    $inner->where('area', $property->area)
                        ->where('bedrooms', $property->bedrooms);
                });

            if ($property->landmark) {
                $query->orWhere('landmark', $property->landmark);
            }
        })
        ->get();
}

The query looks simple, but there is an important Laravel detail here.

Group your orWhere clauses

The exclusion:

->where('id', '!=', $property->id)

needs to apply to every branch of the candidate query.

That's why the orWhere conditions are wrapped in a closure:

->where(function ($query) {
    // OR conditions
});

Without that grouping, you can accidentally generate logic equivalent to:

id != ?
AND fingerprint = ?
OR area = ?

The OR can escape the intended grouping and return the current property itself.

For queries with multiple OR conditions, explicit grouping is worth paying attention to.

---

3. Scoring the Candidates

Once I have a relatively small candidate set, I score each property against the new listing.

The scoring model is intentionally simple.

Signal	Points
Same area	25
Same landmark	20
Same bedrooms	15
Rent within 10%	20
Description similarity	20
Maximum	100

The goal isn't to produce a statistically perfect probability.

The goal is to produce a number that a reviewer can understand.

---

Keeping the Explanation

Before calculating the score, I create a details structure:

$details = [
    'area_match' => false,
    'landmark_match' => false,
    'bedrooms_match' => false,
    'rent_match' => null,
    'rent_difference_percent' => null,
    'description_similarity' => 0.0,
    'fingerprint_match' => false,
];

This is important because storing only:

confidence_score = 87

doesn't tell us much.

Instead, we can store:

{
    "area_match": true,
    "landmark_match": true,
    "bedrooms_match": true,
    "rent_match": "within_10_percent",
    "rent_difference_percent": 4.2,
    "description_similarity": 87.4,
    "fingerprint_match": false
}

Now the reviewer can understand why the system flagged the properties.

---

4. Comparing the Area

The area comparison is straightforward:

if (strcasecmp(trim($a->area), trim($b->area)) === 0) {
    $details['area_match'] = true;

    $points += self::WEIGHT_AREA;
}

With:

private const WEIGHT_AREA = 25;

---

5. Comparing Rent

Rent shouldn't need to be exactly the same.

Different agents might advertise the same property at slightly different prices.

For example:

Agent A: ₦1,500,000
Agent B: ₦1,550,000

These listings could still represent the same property.

So I calculate the percentage difference:

if ((float) $a->rent > 0 && (float) $b->rent > 0) {
    $diff = abs((float) $a->rent - (float) $b->rent)
        / max((float) $a->rent, (float) $b->rent);

    $details['rent_difference_percent'] = round($diff * 100, 2);

    if ($diff <= 0.10) {
        $details['rent_match'] = 'within_10_percent';

        $points += self::WEIGHT_RENT;
    } elseif ($diff <= 0.20) {
        $details['rent_match'] = 'within_20_percent';

        $points += self::WEIGHT_RENT / 2;
    }
}

So:

• Within 10% → 20 points
• Within 20% → 10 points
• More than 20% → 0 points

This gives us a little tolerance without treating every price as equivalent.

---

6. Comparing Descriptions

For descriptions, I used PHP's built-in similar_text() rather than introducing another package.

similar_text(
    strtolower($a->description),
    strtolower($b->description),
    $sim
);

if ($sim >= 80) {
    $points += self::WEIGHT_DESCRIPTION;
} elseif ($sim >= 60) {
    $points += self::WEIGHT_DESCRIPTION / 2;
}

The thresholds are:

80%+ → 20 points
60–79% → 10 points
<60% → 0 points

This works reasonably well for short rental descriptions.

It is not intended to be a sophisticated natural-language similarity model.

And that's okay for this stage.

---

7. The Fingerprint Rule

A matching fingerprint is a strong signal:

Area
+
Landmark
+
Bedrooms
+
Property type

If those fields all match, I want the listing to be reviewed even if some other signals differ.

The final score is calculated like this:

$confidenceScore = round(min($points, 100), 2);

if ($details['fingerprint_match']) {
    $confidenceScore = max($confidenceScore, 90.0);
}

So a fingerprint match effectively creates a minimum score of 90.

That means the system classifies it as a very likely duplicate.

This is a deliberate product decision rather than a mathematical requirement.

Later, this should probably become configurable because two properties can theoretically share the same area, landmark, bedrooms and type while still being different properties.

---

8. Turning the Score Into a Match Level

I use an enum to map the score into a meaningful state:

public static function fromScore(float $score): self
{
    return match (true) {
        $score >= 90 => self::VERY_LIKELY,
        $score >= 70 => self::POSSIBLE,
        default      => self::UNIQUE,
    };
}

The resulting categories are:

Score	Level
90–100	Very likely
70–89	Possible
<70	Unique

The UI doesn't need to know how the score was calculated.

It simply displays the appropriate match level and the underlying explanation.

---

9. Persisting the Match Details

The score and explanation are stored on a duplicate_matches record.

For example:

property_id
matched_property_id
confidence_score
match_details
status

The match_details column is JSON.

That means I can store the complete scoring breakdown without creating a separate database column for every signal.

For example:

{
    "area_match": true,
    "landmark_match": true,
    "bedrooms_match": true,
    "rent_match": "within_10_percent",
    "rent_difference_percent": 4.2,
    "description_similarity": 87.4,
    "fingerprint_match": true
}

The reviewer UI can then display something like:

Very likely duplicate — 94

✓ Same area
✓ Same landmark
✓ Same bedrooms
✓ Rent within 10%
✓ Similar description
✓ Matching property fingerprint

That's much more useful than simply saying:

Duplicate confidence: 94%

---

10. Reviewer Decisions Must Survive Edits

There is another problem with running detection every time a property changes.

Suppose a reviewer sees:

Property A ↔ Property B

and decides:

These are different properties.

The reviewer rejects the match.

Then the agent edits the listing.

If detect() simply runs again and recreates the match, the reviewer gets the same warning again.

That's frustrating.

So duplicate matches have state:

pending
rejected
confirmed

When checking an existing match:

$existing = DuplicateMatch::query()
    ->where('property_id', $property->id)
    ->where('matched_property_id', $candidate->id)
    ->first();

if ($existing && $existing->status === 'rejected' && ! $rescan) {
    continue;
}

if ($existing && $existing->status === 'confirmed') {
    continue;
}

A rejected match stays rejected during normal edits.

A manual rescan can explicitly override that decision.

This makes the system behave more like a workflow rather than just a calculation.

---

11. Avoiding Timeline Noise

Duplicate detection can run frequently.

If every score recalculation creates a timeline event, the property history becomes noisy.

So I only log meaningful changes.

For example:

$scoreChanged = $existing
    && abs(
        (float) $existing->confidence_score
        - $score->confidenceScore
    ) >= 5;

A timeline event is created when:

• A new match is detected
• A manual rescan happens
• The score changes significantly
• A reviewer confirms/rejects a match

This keeps the audit trail useful.

---

12. Where Detection Runs

The detector runs after the property has been saved.

For example:

// PropertyCreator

$this->duplicateDetection->detect(
    $property,
    $user
);

And after an update:

// PropertyController@update

$this->duplicateDetection->detect(
    $property->fresh(),
    $user
);

This is also useful for bulk imports.

My .xlsx importer reuses the same property creation workflow.

That means imported properties automatically go through duplicate detection without implementing another version of the algorithm.

---

13. Testing the Detector

I didn't just test properties that I expected to be duplicates.

I created test spreadsheets containing both duplicates and controls.

Some examples:

Scenario	Score
Same fingerprint + cosmetic differences	100
Same area + different landmark + similar rent/description	80
Duplicate properties in one import	100
Same area but different size	25
Same area + bedrooms but different rent/description	40

The control cases are extremely important.

Without them, it's easy to keep increasing the number of signals until almost every property gets flagged.

A duplicate detector isn't useful if reviewers don't trust its results.

---

14. What I'd Fix Next

The current implementation works, but there are several areas I would improve.

Canonical Areas and Landmarks

Exact string matching creates problems:

Yaba

and:

Yaba, Lagos

are treated as different.

Similarly:

Opp. Yaba Tech

and:

Opposite Yaba Tech

are different strings.

A better approach would be a canonical area/landmark system with aliases.

For example:

Yaba
├── Yaba, Lagos
├── Yaba Lagos
└── Yaba Mainland

could all resolve to the same canonical identifier.

---

Case-Sensitive Candidate Queries

The scoring logic uses:

strcasecmp()

but the candidate query uses normal SQL equality.

That creates an inconsistency.

For example:

Yaba

and:

yaba

may score as a match but never become candidates in the first place.

With PostgreSQL, normalized columns or LOWER() expression indexes would make this more consistent.

---

Bedroom Type Casting

The database may return:

2

while another value might be:

"2"

If strict comparisons are used, those can behave differently.

Casting the model attribute is safer:

protected $casts = [
    'bedrooms' => 'integer',
];

---

Normalizing Match Pairs

Currently:

A → B

and:

B → A

can become separate records.

That means reviewers could potentially see the same duplicate pair twice.

A better approach is to normalize the IDs:

$firstId = min($property->id, $candidate->id);
$secondId = max($property->id, $candidate->id);

Then always store:

first_id
second_id

in a consistent order.

---

similar_text() Performance

PHP's similar_text() is convenient, but it isn't something I'd want to run against large amounts of text indefinitely.

For short rental descriptions, it's acceptable.

If descriptions become long or the candidate set grows significantly, I'd add a cheaper prefilter before running the similarity calculation.

---

The Fingerprint Floor

Currently:

if ($details['fingerprint_match']) {
    $confidenceScore = max($confidenceScore, 90);
}

This means matching identity fields can result in a duplicate score of 90 even when the rent is very different.

Sometimes that's correct.

Sometimes it isn't.

I'd eventually make the fingerprint floor configurable so product/operations can tune the behavior based on real reviewer decisions.

---

15. Why I Didn't Start With Machine Learning

There are three main reasons.

1. There wasn't enough training data

A new marketplace doesn't have thousands of confirmed duplicate/non-duplicate examples.

Without good labelled data, an ML model isn't automatically going to solve the problem.

2. Reviewers need explanations

If a reviewer asks:

"Why did you flag these two properties?"

I can answer:

"Same area, same landmark, same bedrooms, rent is 4.2% apart, and the descriptions are 87% similar."

That's actionable.

A black-box prediction isn't as useful for an operational workflow.

3. The weights are easy to change

The scoring weights are centralized:

private const WEIGHT_AREA = 25;
private const WEIGHT_LANDMARK = 20;
private const WEIGHT_BEDROOMS = 15;
private const WEIGHT_RENT = 20;
private const WEIGHT_DESCRIPTION = 20;

If reviewer feedback shows that one signal is too aggressive, I can tune it without introducing a new model-training pipeline.

---

The Bigger Lesson

The interesting part of this project wasn't really the scoring formula.

It was the workflow around it.

A useful duplicate detection system needs to answer more than:

"Are these two properties similar?"

It also needs to answer:

• Why were they flagged?
• What happens when a reviewer disagrees?
• Does that decision survive future edits?
• Can the reviewer rescan?
• Can the match be confirmed?
• Can the listings eventually be merged?
• Can we audit what happened?
• Can the scoring rules be changed later?

The detection algorithm is only one part of the feature.

The review workflow is what makes the detection useful in production.

---

Takeaways

If you're building duplicate detection for a marketplace, especially in a market without clean addresses:

1. Don't put a unique constraint on something humans are supposed to review.

2. Narrow candidates in SQL, then perform detailed scoring in application code.

3. Group your orWhere clauses carefully.

4. Store the explanation, not just the score.

5. Treat reviewer decisions as persistent state.

6. Test controls, not only the duplicates you want to catch.

7. Start with deterministic rules you can explain.

8. Use real reviewer decisions to improve the rules over time.

You don't always need machine learning to build useful marketplace intelligence.

Sometimes the best first version is a simple system that understands the signals your market actually has, produces a result people can explain, and gives humans the final say.