Data Engineer (Web Scraping)

Key Responsibilities:

1. Data Scraping and Collection

- Gather procurement requirements from target markets by writing scripts (Python, JavaScript, etc.) or using tools

- Collect enterprise information, product details, and trade data from multiple sources (websites, APIs, databases, etc.)

- Maintain and optimize data scraping scripts to ensure efficiency and stability

- Handle anti-scraping mechanisms to ensure data collection compliance

2. Data Cleaning and Processing

- Clean and format the raw scraped data

- Identify and remove duplicate and invalid data

- Standardize data formats (country, city, category, HS code, etc.)

- Process multilingual data (Chinese, English, Japanese, Thai, etc.)

3. Data Review and Screening

- Verify the authenticity and validity of procurement requirements according to platform standards

- Verify the accuracy and completeness of enterprise information

- Screen high-quality data that meets platform requirements

- Identify and flag suspicious or low-quality data

4. Data Entry and Management

- Store the approved data in the platform database

- Maintain the data classification and labeling system

- Manage data versions and update records

- Ensure the accuracy and completeness of data entry

5. Data Quality Monitoring

- Regularly check data quality metrics (completeness, accuracy, timeliness, etc.)

- Monitor the execution status of data scraping tasks

- Identify and resolve data quality issues

- Generate data quality reports

6. Tool Development and Optimization

- Develop or optimize data scraping tools and scripts

- Establish data review workflows and standards

- Enhance the automation level of data processing

- Optimize data processing efficiency

7. Target Market Research

- Research data sources and acquisition channels in target markets

- Understand the data characteristics and formats of different markets

- Identify new data collection opportunities

- Monitor market changes and adjust data collection strategies

II. Job Requirements:

Educational Background:

- Bachelor's degree or above in Computer Science, Data Science, Information Technology, or related fields
Work Experience:

- 1-3 years of experience in data scraping, data processing, or data analysis

- Experience in web scraping development is preferred

- Experience in data cleaning and auditing is preferred

- Experience in B2B platforms or trade data is preferred

- Candidates with cross-border trade industry background are preferred

Technical Skills:

- Programming Languages: Proficient in Python (required) familiarity with JavaScript, Java, etc. is preferred

- Web Scraping Frameworks: Familiar with Scrapy, BeautifulSoup, Selenium, Playwright, and other scraping tools

- Data Processing: Familiar with Pandas, NumPy, and other data processing libraries

- Database: Familiar with MySQL, PostgreSQL, MongoDB, and other database operations

- API: Understanding of RESTful API and JSON data processing

- Tools: Familiar with Git version control, Jupyter Notebook, and other development tools

- Regular Expressions: Able to use regular expressions for text matching and processing

- Multilingual Processing: Experience in multilingual text processing (Chinese, English, Japanese, Thai, etc.) is preferred

Business Skills:

- Understanding of B2B cross-border trade business processes

- Familiar with the data structure of Request for Quotation (RFQ)

- Understanding of trade-related concepts such as company information, product categories, and HS codes

- Capable of assessing the authenticity and validity of data

Soft Skills:

- Fluent in English and Mandarin (spoken and written)

- Detail-oriented and meticulous, capable of handling large volumes of repetitive tasks

- Excellent problem-solving skills and logical thinking

- Strong learning ability and adaptability

- Capable of working under pressure and completing tasks on time

- Strong teamwork spirit

Nice to Have:

- Project experience with web scraping frameworks such as Scrapy and Selenium

- Experience in data cleaning and ETL

- Understanding of machine learning or natural language processing

- Proficient in Docker and Linux systems

Experience in data visualization (Tableau, Power BI, etc.)

Please send your resume to [Confidential Information]