# AlfaMall PHP Scraper

Standalone PHP CLI scraper for collecting AlfaMall products and brands.

The site is a Next.js app. Listing pages include product JSON in the rendered HTML, and product details are available from `https://be.alfamall.com/storefront/product/{handle}?storefront=23`.

## Requirements

- PHP 8.0+
- PHP cURL extension

## Usage

```bash
php scraper.php --start=https://alfamall.com/electronics --max-pages=1
```

Fetch richer detail records for each discovered product:

```bash
php scraper.php --start=https://alfamall.com/electronics --max-pages=1 --details=1
```

Crawl multiple starting pages:

```bash
php scraper.php --start=https://alfamall.com/electronics,https://alfamall.com/haier-official-store --max-pages=5 --details=1
```

## Outputs

Files are written to `output/` by default:

- `products.json`
- `products.csv`
- `brands.json`
- `brands.csv`

## Useful Options

- `--start=URL[,URL]`: comma-separated start URLs.
- `--max-pages=N`: cap the number of AlfaMall pages visited.
- `--details=1`: call the backend detail endpoint for each product.
- `--storefront=23`: Pakistan/PKR storefront id found from AlfaMall's public store list.
- `--delay-ms=700`: delay between requests.
- `--output=DIR`: custom output directory.
- `--write-output=0`: skip CSV/JSON files when using the scraper class from the web/database path.

Keep `--max-pages` and `--delay-ms` conservative. AlfaMall's `robots.txt` currently allows crawling, but you should still avoid aggressive request rates and use the data only in ways that respect the site's terms.

## Web Front

The web front is in `public/index.php` and is intended to be served as:

```text
scrapper.ecompk.com
```

It requires a password before any page or download is available. Scrape results are stored in MySQL/MariaDB tables. CSV/JSON downloads are generated from the database through authenticated PHP only.

Create the password config:

```bash
cd /var/www/scrapper.ecompk.com
read -s -p "Scrapper password: " SCRAPPER_PASS; echo
HASH=$(SCRAPPER_PASS="$SCRAPPER_PASS" php -r 'echo password_hash(getenv("SCRAPPER_PASS"), PASSWORD_DEFAULT);')
sudo tee config.php >/dev/null <<PHP
<?php
declare(strict_types=1);

return [
    'password_hash' => '$HASH',
    'db' => [
        'dsn' => 'mysql:host=127.0.0.1;dbname=ecompk_scrapper;charset=utf8mb4',
        'user' => 'ecompk_scrapper',
        'password' => 'CHANGE_DB_PASSWORD',
    ],
];
PHP
unset SCRAPPER_PASS HASH
```

## Existing Apache + Remote RDS Deployment

Run these on the server after pointing DNS `scrapper.ecompk.com` to the server IP and uploading this project to `/var/www/scrapper.ecompk.com`. These commands assume Apache/PHP are already installed.

```bash
sudo find /var/www/scrapper.ecompk.com -type d -exec chmod 755 {} \;
sudo find /var/www/scrapper.ecompk.com -type f -exec chmod 644 {} \;
```

Create `/var/www/scrapper.ecompk.com/config.php` with your RDS database details:

```bash
cd /var/www/scrapper.ecompk.com
read -s -p "Scrapper login password: " SCRAPPER_PASS; echo
HASH=$(SCRAPPER_PASS="$SCRAPPER_PASS" php -r 'echo password_hash(getenv("SCRAPPER_PASS"), PASSWORD_DEFAULT);')

sudo tee config.php >/dev/null <<PHP
<?php
declare(strict_types=1);

return [
    'password_hash' => '$HASH',
    'db' => [
        'dsn' => 'mysql:host=database-1.cbky4okkya93.us-east-1.rds.amazonaws.com;port=3306;dbname=scrapper;charset=utf8mb4',
        'user' => 'hbl3',
        'password' => 'PASTE_RDS_PASSWORD_HERE',
    ],
];
PHP
unset SCRAPPER_PASS HASH
```

Create/update the database tables without the `mysql` command:

```bash
php /var/www/scrapper.ecompk.com/setup_db.php
```

Apache virtual host:

```bash
sudo tee /etc/apache2/sites-available/scrapper.ecompk.com.conf >/dev/null <<'APACHE'
<VirtualHost *:80>
    ServerName scrapper.ecompk.com
    DocumentRoot /var/www/scrapper.ecompk.com/public

    <Directory /var/www/scrapper.ecompk.com/public>
        AllowOverride All
        Require all granted
        Options -Indexes
    </Directory>

    <Directory /var/www/scrapper.ecompk.com>
        Require all denied
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/scrapper.ecompk.com-error.log
    CustomLog ${APACHE_LOG_DIR}/scrapper.ecompk.com-access.log combined
}
APACHE

sudo a2ensite scrapper.ecompk.com.conf
sudo apache2ctl configtest
sudo systemctl reload apache2
```

For a quick local test:

```bash
php -S 127.0.0.1:8080 -t public
```