增加抓取producthunt的数据
This commit is contained in:
124
integrated_scraper.log
Normal file
124
integrated_scraper.log
Normal file
@@ -0,0 +1,124 @@
|
||||
2025-11-22 22:20:16.912 | INFO | __main__:run_test:201 - === 开始整合版本测试 ===
|
||||
2025-11-22 22:20:16.912 | INFO | __main__:init_product_database:62 - 正在初始化测试数据库...
|
||||
2025-11-22 22:20:16.918 | SUCCESS | __main__:init_product_database:85 - 测试数据库初始化完成
|
||||
2025-11-22 22:20:16.918 | INFO | __main__:query_producthunt_urls:41 - 正在查询tophub_data.db数据库...
|
||||
2025-11-22 22:20:16.919 | SUCCESS | __main__:query_producthunt_urls:53 - 找到 3 个包含producthunt.com的链接
|
||||
2025-11-22 22:20:16.919 | INFO | __main__:run_test:212 - 找到 3 个ProductHunt链接
|
||||
2025-11-22 22:20:16.921 | INFO | __main__:run_test:217 - 处理URL: https://www.producthunt.com/products/pixley-ai
|
||||
2025-11-22 22:20:16.922 | INFO | __main__:scrape_product_info:164 - 开始抓取: https://www.producthunt.com/products/pixley-ai
|
||||
2025-11-22 22:20:16.922 | INFO | playwright_get_data:connect_to_existing_chrome:30 - 正在连接到Chrome远程调试端口 9222
|
||||
2025-11-22 22:20:17.558 | SUCCESS | playwright_get_data:connect_to_existing_chrome:57 - 成功连接到Chrome浏览器
|
||||
2025-11-22 22:20:17.558 | INFO | playwright_get_data:navigate_to_producthunt:111 - 正在访问: https://www.producthunt.com/products/pixley-ai
|
||||
2025-11-22 22:20:18.566 | INFO | playwright_get_data:navigate_to_producthunt:116 - 等待页面标题包含'Product Hunt'...
|
||||
2025-11-22 22:20:18.574 | INFO | playwright_get_data:navigate_to_producthunt:124 - 当前页面标题: Pixley AI: Pixley lets kids turn their ideas into cartoons in minutes | Product Hunt
|
||||
2025-11-22 22:20:18.574 | SUCCESS | playwright_get_data:navigate_to_producthunt:128 - 页面标题已包含'Product Hunt'，等待时间: 0秒
|
||||
2025-11-22 22:20:18.575 | SUCCESS | playwright_get_data:navigate_to_producthunt:129 - Product Hunt网站已成功打开
|
||||
2025-11-22 22:20:18.575 | INFO | playwright_get_data:extract_product_info:291 - 正在提取产品名称...
|
||||
2025-11-22 22:20:18.575 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 产品名称 - 选择器: //h1
|
||||
2025-11-22 22:20:18.601 | INFO | playwright_get_data:extract_product_info:297 - 产品名称: Pixley AI
|
||||
2025-11-22 22:20:18.601 | INFO | playwright_get_data:extract_product_info:304 - 正在提取产品简介...
|
||||
2025-11-22 22:20:18.602 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 产品简介 - 选择器: //*[@class="relative text-16 font-normal text-gray-700"]//div
|
||||
2025-11-22 22:20:18.607 | INFO | playwright_get_data:extract_product_info:310 - 产品简介: Pixley is the first platform that lets children turn their drawings and ideas into personalized, animated cartoons in minutes. Until now, making animation was slow, expensive, and impossible to person...
|
||||
2025-11-22 22:20:18.608 | INFO | playwright_get_data:extract_product_info:317 - 正在提取用户数...
|
||||
2025-11-22 22:20:18.608 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 用户数 - 选择器: //*[@class="flex flex-row gap-2"]//div/div[2]/span/p
|
||||
2025-11-22 22:20:18.612 | INFO | playwright_get_data:extract_product_info:323 - 用户数: 53 followers
|
||||
2025-11-22 22:20:18.613 | INFO | playwright_get_data:extract_product_info:330 - 正在提取制作人发言链接...
|
||||
2025-11-22 22:20:18.613 | INFO | playwright_get_data:extract_product_info:333 - 等待页面元素加载...
|
||||
2025-11-22 22:20:38.616 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 制作人span标签 - 选择器: //span[contains(@class, "absolute")]
|
||||
2025-11-22 22:20:38.624 | WARNING | playwright_get_data:extract_product_info:370 - 未找到XPath为//span[contains(@class, "absolute")]的元素
|
||||
2025-11-22 22:20:38.626 | INFO | playwright_get_data:extract_product_info:384 - 产品信息已保存到临时文件: temp_product_info.txt
|
||||
2025-11-22 22:20:38.920 | INFO | playwright_get_data:extract_product_info:389 - 页面截图已保存到: product_screenshot.png
|
||||
2025-11-22 22:20:38.920 | SUCCESS | __main__:scrape_product_info:186 - 成功提取产品信息: Pixley AI
|
||||
2025-11-22 22:20:38.925 | INFO | playwright_get_data:close:401 - 浏览器连接已关闭
|
||||
2025-11-22 22:20:38.935 | INFO | playwright_get_data:close:405 - Playwright实例已关闭
|
||||
2025-11-22 22:20:38.938 | INFO | __main__:save_product_info:151 - 新增产品信息: Pixley AI
|
||||
2025-11-22 22:20:38.941 | SUCCESS | __main__:run_test:232 - 成功保存产品信息: Pixley AI
|
||||
2025-11-22 22:20:38.942 | INFO | __main__:run_test:217 - 处理URL: https://www.producthunt.com/products/burner-2
|
||||
2025-11-22 22:20:38.942 | INFO | __main__:scrape_product_info:164 - 开始抓取: https://www.producthunt.com/products/burner-2
|
||||
2025-11-22 22:20:38.942 | INFO | playwright_get_data:connect_to_existing_chrome:30 - 正在连接到Chrome远程调试端口 9222
|
||||
2025-11-22 22:20:39.570 | SUCCESS | playwright_get_data:connect_to_existing_chrome:57 - 成功连接到Chrome浏览器
|
||||
2025-11-22 22:20:39.571 | INFO | playwright_get_data:navigate_to_producthunt:111 - 正在访问: https://www.producthunt.com/products/burner-2
|
||||
2025-11-22 22:20:41.223 | INFO | playwright_get_data:navigate_to_producthunt:116 - 等待页面标题包含'Product Hunt'...
|
||||
2025-11-22 22:20:41.232 | INFO | playwright_get_data:navigate_to_producthunt:124 - 当前页面标题: Burner: A low cost wallet to gift crypto | Product Hunt
|
||||
2025-11-22 22:20:41.232 | SUCCESS | playwright_get_data:navigate_to_producthunt:128 - 页面标题已包含'Product Hunt'，等待时间: 0秒
|
||||
2025-11-22 22:20:41.232 | SUCCESS | playwright_get_data:navigate_to_producthunt:129 - Product Hunt网站已成功打开
|
||||
2025-11-22 22:20:41.232 | INFO | playwright_get_data:extract_product_info:291 - 正在提取产品名称...
|
||||
2025-11-22 22:20:41.232 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 产品名称 - 选择器: //h1
|
||||
2025-11-22 22:20:41.261 | INFO | playwright_get_data:extract_product_info:297 - 产品名称: Burner
|
||||
2025-11-22 22:20:41.261 | INFO | playwright_get_data:extract_product_info:304 - 正在提取产品简介...
|
||||
2025-11-22 22:20:41.261 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 产品简介 - 选择器: //*[@class="relative text-16 font-normal text-gray-700"]//div
|
||||
2025-11-22 22:20:41.268 | INFO | playwright_get_data:extract_product_info:310 - 产品简介: Burner is a small, secure computer that keeps your digital assets safe. It’s based on the same secure chips used in big, complicated and expensive hardware wallets. However, unlike hardware wallets, y...
|
||||
2025-11-22 22:20:41.268 | INFO | playwright_get_data:extract_product_info:317 - 正在提取用户数...
|
||||
2025-11-22 22:20:41.268 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 用户数 - 选择器: //*[@class="flex flex-row gap-2"]//div/div[2]/span/p
|
||||
2025-11-22 22:20:41.273 | INFO | playwright_get_data:extract_product_info:323 - 用户数: 305 followers
|
||||
2025-11-22 22:20:41.273 | INFO | playwright_get_data:extract_product_info:330 - 正在提取制作人发言链接...
|
||||
2025-11-22 22:20:41.273 | INFO | playwright_get_data:extract_product_info:333 - 等待页面元素加载...
|
||||
2025-11-22 22:21:01.288 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 制作人span标签 - 选择器: //span[contains(@class, "absolute")]
|
||||
2025-11-22 22:21:01.292 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 制作人链接 - 选择器: //span[contains(@class, "absolute")]/parent::a
|
||||
2025-11-22 22:21:01.300 | INFO | playwright_get_data:extract_product_info:363 - 制作人链接: https://www.producthunt.com/products/burner-2/launches/burner-terminal
|
||||
2025-11-22 22:21:01.300 | INFO | playwright_get_data:record_click:75 - 记录点击: - 坐标(制作人链接, 点击制作人链接在当前窗口打开) - 选择器:
|
||||
2025-11-22 22:21:01.300 | INFO | playwright_get_data:extract_maker_statement_from_current_window:169 - 正在在当前窗口打开制作人链接: https://www.producthunt.com/products/burner-2/launches/burner-terminal
|
||||
2025-11-22 22:21:04.256 | INFO | playwright_get_data:extract_maker_statement_from_current_window:176 - 等待title元素出现并包含产品名称（最长等待2分钟）...
|
||||
2025-11-22 22:23:04.279 | ERROR | playwright_get_data:extract_maker_statement_from_current_window:194 - 等待title元素失败: Page.wait_for_selector: Timeout 120000ms exceeded.
|
||||
Call log:
|
||||
- waiting for locator("title") to be visible
|
||||
240 × locator resolved to hidden <title>Burner Terminal - Tap to Pay for Stablecoins | Pr…</title>
|
||||
|
||||
2025-11-22 22:23:04.279 | INFO | playwright_get_data:extract_maker_statement_from_current_window:197 - 再等待30秒，确保页面完全加载...
|
||||
2025-11-22 22:23:34.282 | INFO | playwright_get_data:extract_maker_statement_from_current_window:201 - 正在提取制作人评论内容...
|
||||
2025-11-22 22:23:34.287 | WARNING | playwright_get_data:extract_maker_statement_from_current_window:213 - 未找到XPath为//*[@id="comment-4597755"]/div/div[2]/div/div/div的元素
|
||||
2025-11-22 22:23:34.288 | INFO | playwright_get_data:extract_product_info:384 - 产品信息已保存到临时文件: temp_product_info.txt
|
||||
2025-11-22 22:23:34.507 | INFO | playwright_get_data:extract_product_info:389 - 页面截图已保存到: product_screenshot.png
|
||||
2025-11-22 22:23:34.507 | SUCCESS | __main__:scrape_product_info:186 - 成功提取产品信息: Burner
|
||||
2025-11-22 22:23:34.511 | INFO | playwright_get_data:close:401 - 浏览器连接已关闭
|
||||
2025-11-22 22:23:34.521 | INFO | playwright_get_data:close:405 - Playwright实例已关闭
|
||||
2025-11-22 22:23:34.524 | INFO | __main__:save_product_info:151 - 新增产品信息: Burner
|
||||
2025-11-22 22:23:34.527 | SUCCESS | __main__:run_test:232 - 成功保存产品信息: Burner
|
||||
2025-11-22 22:23:34.527 | INFO | __main__:run_test:217 - 处理URL: https://www.producthunt.com/products/american-ratings-lead-magnet-portal
|
||||
2025-11-22 22:23:34.528 | INFO | __main__:scrape_product_info:164 - 开始抓取: https://www.producthunt.com/products/american-ratings-lead-magnet-portal
|
||||
2025-11-22 22:23:34.528 | INFO | playwright_get_data:connect_to_existing_chrome:30 - 正在连接到Chrome远程调试端口 9222
|
||||
2025-11-22 22:23:35.209 | SUCCESS | playwright_get_data:connect_to_existing_chrome:57 - 成功连接到Chrome浏览器
|
||||
2025-11-22 22:23:35.209 | INFO | playwright_get_data:navigate_to_producthunt:111 - 正在访问: https://www.producthunt.com/products/american-ratings-lead-magnet-portal
|
||||
2025-11-22 22:23:38.225 | INFO | playwright_get_data:navigate_to_producthunt:116 - 等待页面标题包含'Product Hunt'...
|
||||
2025-11-22 22:23:38.237 | INFO | playwright_get_data:navigate_to_producthunt:124 - 当前页面标题: American Ratings Lead Magnet Portal: Get Your Verified A-I-R-S Number & Boost Global Credibility | Product Hunt
|
||||
2025-11-22 22:23:38.237 | SUCCESS | playwright_get_data:navigate_to_producthunt:128 - 页面标题已包含'Product Hunt'，等待时间: 0秒
|
||||
2025-11-22 22:23:38.237 | SUCCESS | playwright_get_data:navigate_to_producthunt:129 - Product Hunt网站已成功打开
|
||||
2025-11-22 22:23:38.237 | INFO | playwright_get_data:extract_product_info:291 - 正在提取产品名称...
|
||||
2025-11-22 22:23:38.238 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 产品名称 - 选择器: //h1
|
||||
2025-11-22 22:23:38.369 | INFO | playwright_get_data:extract_product_info:297 - 产品名称: American Ratings Lead Magnet Portal
|
||||
2025-11-22 22:23:38.369 | INFO | playwright_get_data:extract_product_info:304 - 正在提取产品简介...
|
||||
2025-11-22 22:23:38.369 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 产品简介 - 选择器: //*[@class="relative text-16 font-normal text-gray-700"]//div
|
||||
2025-11-22 22:23:38.383 | INFO | playwright_get_data:extract_product_info:310 - 产品简介: Build verified business credibility with the American Ratings Lead Magnet Portal — the trusted platform for authentic verification and global rating credentials. Get your A-I-R-S Number to showcase tr...
|
||||
2025-11-22 22:23:38.383 | INFO | playwright_get_data:extract_product_info:317 - 正在提取用户数...
|
||||
2025-11-22 22:23:38.383 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 用户数 - 选择器: //*[@class="flex flex-row gap-2"]//div/div[2]/span/p
|
||||
2025-11-22 22:23:38.390 | INFO | playwright_get_data:extract_product_info:323 - 用户数: 24 followers
|
||||
2025-11-22 22:23:38.391 | INFO | playwright_get_data:extract_product_info:330 - 正在提取制作人发言链接...
|
||||
2025-11-22 22:23:38.391 | INFO | playwright_get_data:extract_product_info:333 - 等待页面元素加载...
|
||||
2025-11-22 22:23:58.404 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 制作人span标签 - 选择器: //span[contains(@class, "absolute")]
|
||||
2025-11-22 22:23:58.412 | INFO | playwright_get_data:record_dom_selection:86 - 记录DOM选取: 制作人链接 - 选择器: //span[contains(@class, "absolute")]/parent::a
|
||||
2025-11-22 22:23:58.423 | INFO | playwright_get_data:extract_product_info:363 - 制作人链接: https://www.producthunt.com/p/american-ratings-lead-magnet-portal/a-i-r-s-number-american-ratings-lead-magnet-webinar-channel-partner-credit-100k-25m
|
||||
2025-11-22 22:23:58.423 | INFO | playwright_get_data:record_click:75 - 记录点击: - 坐标(制作人链接, 点击制作人链接在当前窗口打开) - 选择器:
|
||||
2025-11-22 22:23:58.423 | INFO | playwright_get_data:extract_maker_statement_from_current_window:169 - 正在在当前窗口打开制作人链接: https://www.producthunt.com/p/american-ratings-lead-magnet-portal/a-i-r-s-number-american-ratings-lead-magnet-webinar-channel-partner-credit-100k-25m
|
||||
2025-11-22 22:24:00.405 | INFO | playwright_get_data:extract_maker_statement_from_current_window:176 - 等待title元素出现并包含产品名称（最长等待2分钟）...
|
||||
2025-11-22 22:26:00.422 | ERROR | playwright_get_data:extract_maker_statement_from_current_window:194 - 等待title元素失败: Page.wait_for_selector: Timeout 120000ms exceeded.
|
||||
Call log:
|
||||
- waiting for locator("title") to be visible
|
||||
239 × locator resolved to hidden <title>A-I-R-S Number - American Ratings Lead Magnet Web…</title>
|
||||
|
||||
2025-11-22 22:26:00.422 | INFO | playwright_get_data:extract_maker_statement_from_current_window:197 - 再等待30秒，确保页面完全加载...
|
||||
2025-11-22 22:26:30.438 | INFO | playwright_get_data:extract_maker_statement_from_current_window:201 - 正在提取制作人评论内容...
|
||||
2025-11-22 22:26:30.446 | WARNING | playwright_get_data:extract_maker_statement_from_current_window:213 - 未找到XPath为//*[@id="comment-4597755"]/div/div[2]/div/div/div的元素
|
||||
2025-11-22 22:26:30.447 | INFO | playwright_get_data:extract_product_info:384 - 产品信息已保存到临时文件: temp_product_info.txt
|
||||
2025-11-22 22:26:30.751 | INFO | playwright_get_data:extract_product_info:389 - 页面截图已保存到: product_screenshot.png
|
||||
2025-11-22 22:26:30.751 | SUCCESS | __main__:scrape_product_info:186 - 成功提取产品信息: American Ratings Lead Magnet Portal
|
||||
2025-11-22 22:26:30.756 | INFO | playwright_get_data:close:401 - 浏览器连接已关闭
|
||||
2025-11-22 22:26:30.766 | INFO | playwright_get_data:close:405 - Playwright实例已关闭
|
||||
2025-11-22 22:26:30.768 | INFO | __main__:save_product_info:151 - 新增产品信息: American Ratings Lead Magnet Portal
|
||||
2025-11-22 22:26:30.771 | SUCCESS | __main__:run_test:232 - 成功保存产品信息: American Ratings Lead Magnet Portal
|
||||
2025-11-22 22:26:30.772 | INFO | __main__:show_test_results:261 - === 测试结果统计 ===
|
||||
2025-11-22 22:26:30.773 | INFO | __main__:show_test_results:262 - 数据库中的产品数量: 3
|
||||
2025-11-22 22:26:30.773 | INFO | __main__:show_test_results:265 - 已抓取的产品:
|
||||
2025-11-22 22:26:30.773 | INFO | __main__:show_test_results:267 - - American Ratings Lead Magnet Portal: https://www.producthunt.com/products/american-ratings-lead-magnet-portal
|
||||
2025-11-22 22:26:30.773 | INFO | __main__:show_test_results:267 - - Burner: https://www.producthunt.com/products/burner-2
|
||||
2025-11-22 22:26:30.773 | INFO | __main__:show_test_results:267 - - Pixley AI: https://www.producthunt.com/products/pixley-ai
|
||||
2025-11-22 22:26:30.773 | SUCCESS | __main__:run_test:243 - === 整合版本测试完成 ===
|
||||
Reference in New Issue
Block a user