Merge pull request #664 from crawlab-team/release

Release

CHANGELOG-zh.md

@@ -1,3 +1,20 @@
+# 0.4.9 (2020-03-31)
+### 功能 / 优化
+- **挑战**. 用户可以完成不同的趣味挑战..
+- **更高级的权限控制**. 更细化的权限管理，例如普通用户只能查看或管理自己的爬虫或项目，而管理用户可以查看或管理所有爬虫或项目.
+- **反馈**. 允许用户发送反馈和评分给 Crawlab 开发组.
+- **更好的主页指标**. 优化主页上的指标展示.
+- **可配置爬虫转化为自定义爬虫**. 用户可以将自己的可配置爬虫转化为 Scrapy 自定义爬虫.
+- **查看定时任务触发的任务**. 允许用户查看定时任务触发的任务. [#648](https://github.com/crawlab-team/crawlab/issues/648)
+- **支持结果去重**. 允许用户配置结果去重. [#579](https://github.com/crawlab-team/crawlab/issues/579)
+- **支持任务重试**. 允许任务重新触发历史任务.
+
+### Bug 修复
+- **CLI 无法在 Windows 上使用**. [#580](https://github.com/crawlab-team/crawlab/issues/580)
+- **重新上传错误**. [#643](https://github.com/crawlab-team/crawlab/issues/643) [#640](https://github.com/crawlab-team/crawlab/issues/640)
+- **上传丢失文件目录**. [#646](https://github.com/crawlab-team/crawlab/issues/646)
+- **无法在爬虫定时任务标签中添加定时任务**.
+
 # 0.4.8 (2020-03-11)
 ### 功能 / 优化
 - **支持更多编程语言安装**. 现在用户可以安装或预装更多的编程语言，包括 Java、.Net Core、PHP.

CHANGELOG.md

@@ -1,3 +1,20 @@
+# 0.4.9 (2020-03-31)
+### Features / Enhancement
+- **Challenges**. Users can achieve different challenges based on their actions.
+- **More Advanced Access Control**. More granular access control, e.g. normal users can only view/manage their own spiders/projects and admin users can view/manage all spiders/projects.
+- **Feedback**. Allow users to send feedbacks and ratings to Crawlab team.
+- **Better Home Page Metrics**. Optimized metrics display on home page.
+- **Configurable Spiders Converted to Customized Spiders**. Allow users to convert their configurable spiders into customized spiders which are also Scrapy spiders.
+- **View Tasks Triggered by Schedule**. Allow users to view tasks triggered by a schedule. [#648](https://github.com/crawlab-team/crawlab/issues/648)
+- **Support Results De-Duplication**. Allow users to configure de-duplication of results. [#579](https://github.com/crawlab-team/crawlab/issues/579)
+- **Support Task Restart**. Allow users to re-run historical tasks.
+
+### Bug Fixes
+- **CLI unable to use on Windows**. [#580](https://github.com/crawlab-team/crawlab/issues/580)
+- **Re-upload error**. [#643](https://github.com/crawlab-team/crawlab/issues/643) [#640](https://github.com/crawlab-team/crawlab/issues/640)
+- **Upload missing folders**. [#646](https://github.com/crawlab-team/crawlab/issues/646)
+- **Unable to add schedules in Spider Page**.
+
 # 0.4.8 (2020-03-11)
 ### Features / Enhancement
 - **Support Installations of More Programming Languages**. Now users can install or pre-install more programming languages including Java, .Net Core and PHP.

README-zh.md

@@ -305,6 +305,9 @@ Crawlab使用起来很方便，也很通用，可以适用于几乎任何主流
 <a href="https://github.com/duanbin0414">
   <img src="https://avatars3.githubusercontent.com/u/50389867?s=460&v=4" height="80">
 </a>
+<a href="https://github.com/zkqiang">
+  <img src="https://avatars3.githubusercontent.com/u/32983588?s=460&u=83082ddc0a3020279374b94cce70f1aebb220b3d&v=4" height="80">
+</a>
 
 ## 社区 & 赞助
 

README.md

@@ -272,6 +272,9 @@ Crawlab is easy to use, general enough to adapt spiders in any language and any
 <a href="https://github.com/duanbin0414">
   <img src="https://avatars3.githubusercontent.com/u/50389867?s=460&v=4" height="80">
 </a>
+<a href="https://github.com/zkqiang">
+  <img src="https://avatars3.githubusercontent.com/u/32983588?s=460&u=83082ddc0a3020279374b94cce70f1aebb220b3d&v=4" height="80">
+</a>
 
 ## Community & Sponsorship
 

backend/conf/config.yml

@@ -38,7 +38,7 @@ task:
   workers: 4
 other:
   tmppath: "/tmp"
-version: 0.4.8
+version: 0.4.9
 setting:
   allowRegister: "N"
   enableTutorial: "N"

backend/constants/action.go

@@ -0,0 +1,8 @@
+package constants
+
+const (
+	ActionTypeVisit          = "visit"
+	ActionTypeInstallDep     = "install_dep"
+	ActionTypeInstallLang    = "install_lang"
+	ActionTypeViewDisclaimer = "view_disclaimer"
+)

backend/constants/auth.go

@@ -0,0 +1,7 @@
+package constants
+
+const (
+	OwnerTypeAll    = "all"
+	OwnerTypeMe     = "me"
+	OwnerTypePublic = "public"
+)

backend/constants/challenge.go

@@ -0,0 +1,20 @@
+package constants
+
+const (
+	ChallengeLogin7d                  = "login_7d"
+	ChallengeLogin30d                 = "login_30d"
+	ChallengeLogin90d                 = "login_90d"
+	ChallengeLogin180d                = "login_180d"
+	ChallengeCreateCustomizedSpider   = "create_customized_spider"
+	ChallengeCreateConfigurableSpider = "create_configurable_spider"
+	ChallengeCreateSchedule           = "create_schedule"
+	ChallengeCreateNodes              = "create_nodes"
+	ChallengeCreateUser         = "create_user"
+	ChallengeRunRandom                = "run_random"
+	ChallengeScrape1k                 = "scrape_1k"
+	ChallengeScrape10k                = "scrape_10k"
+	ChallengeScrape100k               = "scrape_100k"
+	ChallengeInstallDep               = "install_dep"
+	ChallengeInstallLang              = "install_lang"
+	ChallengeViewDisclaimer           = "view_disclaimer"
+)

backend/data/challenge_data.json

@@ -0,0 +1,142 @@
+[
+  {
+    "name": "login_7d",
+    "title_cn": "连续登录 7 天",
+    "title_en": "Logged-in for 7 days",
+    "description_cn": "连续 7 天登录 Crawlab，即可完成挑战！",
+    "description_en": "Logged-in for consecutive 7 days to complete the challenge",
+    "difficulty": 1
+  },
+  {
+    "name": "login_30d",
+    "title_cn": "连续登录 30 天",
+    "title_en": "Logged-in for 30 days",
+    "description_cn": "连续 30 天登录 Crawlab，即可完成挑战！",
+    "description_en": "Logged-in for consecutive 30 days to complete the challenge",
+    "difficulty": 2
+  },
+  {
+    "name": "login_90d",
+    "title_cn": "连续登录 90 天",
+    "title_en": "Logged-in for 90 days",
+    "description_cn": "连续 90 天登录 Crawlab，即可完成挑战！",
+    "description_en": "Logged-in for consecutive 90 days to complete the challenge",
+    "difficulty": 3
+  },
+  {
+    "name": "login_180d",
+    "title_cn": "连续登录 180 天",
+    "title_en": "Logged-in for 180 days",
+    "description_cn": "连续 180 天登录 Crawlab，即可完成挑战！",
+    "description_en": "Logged-in for consecutive 180 days to complete the challenge",
+    "difficulty": 4
+  },
+  {
+    "name": "create_customized_spider",
+    "title_cn": "创建 1 个自定义爬虫",
+    "title_en": "Create a customized spider",
+    "description_cn": "在爬虫列表中，点击 '添加爬虫'，选择 '自定义爬虫'，输入相应的参数，点击添加，即可完成挑战！",
+    "description_en": "In Spider List page, click 'Add Spider', select 'Customized Spider', enter params, click 'Add' to finish the challenge.",
+    "difficulty": 1,
+    "path": "/spiders"
+  },
+  {
+    "name": "create_configurable_spider",
+    "title_cn": "创建 1 个可配置爬虫",
+    "title_en": "Create a configurable spider",
+    "description_cn": "在爬虫列表中，点击 '添加爬虫'，选择 '可配置爬虫'，输入相应的参数，点击添加，即可完成挑战！",
+    "description_en": "In Spider List page, click 'Add Spider', select 'Configurable Spider', enter params, click 'Add' to finish the challenge.",
+    "difficulty": 1,
+    "path": "/spiders"
+  },
+  {
+    "name": "run_random",
+    "title_cn": "用随机模式成功运行爬虫",
+    "title_en": "Run a spider in random mode successfully",
+    "description_cn": "在您创建好的爬虫中，导航到其对应的详情页（爬虫列表中点击爬虫），选择随机模式运行一个爬虫，并能运行成功。",
+    "description_en": "In your created spiders, navigate to corresponding detail page (click spider in Spider List page), run a spider in random mode successfully.",
+    "difficulty": 1,
+    "path": "/spiders"
+  },
+  {
+    "name": "scrape_1k",
+    "title_cn": "抓取 1 千条数据",
+    "title_en": "Scrape 1k records",
+    "description_cn": "运行您创建好的爬虫，抓取 1 千条及以上的结果数据，即可完成挑战！",
+    "description_en": "Run your created spiders, scrape 1k and more results to finish the challenge.",
+    "difficulty": 2,
+    "path": "/spiders"
+  },
+  {
+    "name": "scrape_10k",
+    "title_cn": "抓取 1 万条数据",
+    "title_en": "Scrape 10k records",
+    "description_cn": "运行您创建好的爬虫，抓取 1 万条及以上的结果数据，即可完成挑战！",
+    "description_en": "Run your created spiders, scrape 10k and more results to finish the challenge.",
+    "difficulty": 3,
+    "path": "/spiders"
+  },
+  {
+    "name": "scrape_100k",
+    "title_cn": "抓取 10 万条数据",
+    "title_en": "Scrape 100k records",
+    "description_cn": "运行您创建好的爬虫，抓取 10 万条及以上的结果数据，即可完成挑战！",
+    "description_en": "Run your created spiders, scrape 100k and more results to finish the challenge.",
+    "difficulty": 4,
+    "path": "/spiders"
+  },
+  {
+    "name": "create_schedule",
+    "title_cn": "创建 1 个定时任务",
+    "title_en": "Create a schedule",
+    "description_cn": "在定时任务列表中，创建一个定时任务，正确设置好 Cron 表达式，即可完成挑战！",
+    "description_en": "In Schedule List page, create a schedule and configure cron expression to finish the task.",
+    "difficulty": 1,
+    "path": "/schedules"
+  },
+  {
+    "name": "create_nodes",
+    "title_cn": "创建 1 个节点集群",
+    "title_en": "Create a node cluster",
+    "description_cn": "按照文档的部署指南，部署含有 3 个节点的集群，即可完成挑战！",
+    "description_en": "Deploy a 3-node cluster according to the deployment guidance in documentation to finish the task.",
+    "difficulty": 3,
+    "path": "/nodes"
+  },
+  {
+    "name": "install_dep",
+    "title_cn": "安装 1 个依赖",
+    "title_en": "Install a dependency successfully",
+    "description_cn": "在 '节点列表->安装' 或 '节点详情->安装' 中，搜索并安装所需的 1 个依赖，即可完成挑战！",
+    "description_en": "In 'Node List -> Installation' or 'Node Detail -> Installation', search and install a dependency.",
+    "difficulty": 3,
+    "path": "/nodes"
+  },
+  {
+    "name": "install_lang",
+    "title_cn": "安装 1 个语言环境",
+    "title_en": "Install a language successfully",
+    "description_cn": "在 '节点列表->安装' 或 '节点详情->安装' 中，点击安装所需的 1 个语言环境，即可完成挑战！",
+    "description_en": "In 'Node List -> Installation' or 'Node Detail -> Installation', install a language.",
+    "difficulty": 3,
+    "path": "/nodes"
+  },
+  {
+    "name": "view_disclaimer",
+    "title_cn": "阅读免责声明",
+    "title_en": "View disclaimer",
+    "description_cn": "在左侧菜单栏，点击 '免责声明' 查看其内容，即可完成挑战！",
+    "description_en": "In the left side menu, click 'Disclaimer' and view its content to finish the challenge.",
+    "difficulty": 1,
+    "path": "/disclaimer"
+  },
+  {
+    "name": "create_user",
+    "title_cn": "创建 1 个用户",
+    "title_en": "Create a user",
+    "description_cn": "在用户管理页面中创建一个新用户，即可完成挑战！",
+    "description_en": "In User Admin page, create a new user to finish the challenge.",
+    "difficulty": 1,
+    "path": "/users"
+  }
+]

backend/entity/rpc.go

@@ -1,11 +1,11 @@
 package entity
 
 type RpcMessage struct {
-	Id      string            `json:"id"`
-	Method  string            `json:"method"`
-	NodeId  string            `json:"node_id"`
-	Params  map[string]string `json:"params"`
-	Timeout int               `json:"timeout"`
-	Result  string            `json:"result"`
-	Error   string            `json:"error"`
+	Id      string            `json:"id"`      // 消息ID
+	Method  string            `json:"method"`  // 消息方法
+	NodeId  string            `json:"node_id"` // 节点ID
+	Params  map[string]string `json:"params"`  // 参数
+	Timeout int               `json:"timeout"` // 超时
+	Result  string            `json:"result"`  // 结果
+	Error   string            `json:"error"`   // 错误
 }

backend/main.go

@@ -9,6 +9,7 @@ import (
 	"crawlab/model"
 	"crawlab/routes"
 	"crawlab/services"
+	"crawlab/services/challenge"
 	"crawlab/services/rpc"
 	"github.com/apex/log"
 	"github.com/gin-gonic/gin"
@@ -91,6 +92,22 @@ func main() {
 			panic(err)
 		}
 		log.Info("initialized dependency fetcher successfully")
+
+		// 初始化挑战服务
+		if err := challenge.InitChallengeService(); err != nil {
+			log.Error("init challenge service error:" + err.Error())
+			debug.PrintStack()
+			panic(err)
+		}
+		log.Info("initialized challenge service successfully")
+
+		// 初始化清理服务
+		if err := services.InitCleanService(); err != nil {
+			log.Error("init clean service error:" + err.Error())
+			debug.PrintStack()
+			panic(err)
+		}
+		log.Info("initialized clean service successfully")
 	}
 
 	// 初始化任务执行器
@@ -214,6 +231,7 @@ func main() {
 				authGroup.GET("/tasks/:id/log", routes.GetTaskLog)                          // 任务日志
 				authGroup.GET("/tasks/:id/results", routes.GetTaskResults)                  // 任务结果
 				authGroup.GET("/tasks/:id/results/download", routes.DownloadTaskResultsCsv) // 下载任务结果
+				authGroup.POST("/tasks/:id/restart", routes.RestartTask)                    // 重新开始任务
 			}
 			// 定时任务
 			{
@@ -231,6 +249,7 @@ func main() {
 				authGroup.GET("/users/:id", routes.GetUser)       // 用户详情
 				authGroup.POST("/users/:id", routes.PostUser)     // 更改用户
 				authGroup.DELETE("/users/:id", routes.DeleteUser) // 删除用户
+				authGroup.PUT("/users-add", routes.PutUser)       // 添加用户
 				authGroup.GET("/me", routes.GetMe)                // 获取自己账户
 				authGroup.POST("/me", routes.PostMe)              // 修改自己账户
 			}
@@ -254,6 +273,18 @@ func main() {
 				authGroup.POST("/projects/:id", routes.PostProject)     // 新增
 				authGroup.DELETE("/projects/:id", routes.DeleteProject) // 删除
 			}
+			// 挑战
+			{
+				authGroup.GET("/challenges", routes.GetChallengeList)          // 挑战列表
+				authGroup.POST("/challenges-check", routes.CheckChallengeList) // 检查挑战列表
+			}
+			// 操作
+			{
+				//authGroup.GET("/actions", routes.GetActionList)   // 操作列表
+				//authGroup.GET("/actions/:id", routes.GetAction)   // 操作
+				authGroup.PUT("/actions", routes.PutAction) // 新增操作
+				//authGroup.POST("/actions/:id", routes.PostAction) // 修改操作
+			}
 			// 统计数据
 			authGroup.GET("/stats/home", routes.GetHomeStats) // 首页统计数据
 			// 文件
@@ -262,7 +293,7 @@ func main() {
 			authGroup.GET("/git/branches", routes.GetGitRemoteBranches) // 获取 Git 分支
 			authGroup.GET("/git/public-key", routes.GetGitSshPublicKey) // 获取 SSH 公钥
 			authGroup.GET("/git/commits", routes.GetGitCommits)         // 获取 Git Commits
-			authGroup.POST("/git/checkout", routes.PostGitCheckout)         // 获取 Git Commits
+			authGroup.POST("/git/checkout", routes.PostGitCheckout)     // 获取 Git Commits
 		}
 	}
 

backend/mock/spider.go

@@ -1,6 +1,7 @@
 package mock
 
 import (
+	"crawlab/constants"
 	"crawlab/model"
 	"github.com/apex/log"
 	"github.com/gin-gonic/gin"
@@ -26,6 +27,7 @@ var SpiderList = []model.Spider{
 		LastRunTs:   time.Now(),
 		CreateTs:    time.Now(),
 		UpdateTs:    time.Now(),
+		UserId:      constants.ObjectIdNull,
 	},
 }
 

backend/mock/spider_test.go

@@ -2,6 +2,7 @@ package mock
 
 import (
 	"bytes"
+	"crawlab/constants"
 	"crawlab/model"
 	"encoding/json"
 	"github.com/globalsign/mgo/bson"
@@ -61,6 +62,7 @@ func TestPostSpider(t *testing.T) {
 		LastRunTs:   time.Now(),
 		CreateTs:    time.Now(),
 		UpdateTs:    time.Now(),
+		UserId:      constants.ObjectIdNull,
 	}
 	var resp Response
 	var spiderId = "5d429e6c19f7abede924fee2"