-
Notifications
You must be signed in to change notification settings - Fork 0
/
TODOS
30 lines (27 loc) · 1.33 KB
/
TODOS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
- current_task cannot be used if market does not match user languages
[20180201]
- implement bad src generation
- integrate bad ref/src generation into DA task creation
- implement "rank each segment for all systems" behaviour
- check if "all"-"used" batches gives the right data for IWSLT?
[20181024]
- CampaignTeam owner's staff status is never enforced
- Unclear what is best way to model user replacement
* Should a campaign team be the actual generator of user accounts?
* Should there be a replacement action on user level?
- We need a !#$% redesign here...
[20181105]
- Focus on WMT18 release first:
* Should allow to set up 70:10:10:10 and 88:12 data;
* Should support direct and multi modal assessments;
* Should contain instructions on how to setup/run;
* Should contain code to compute results clusters;
* Deadline: 11/11/2018.
- Annotation tasks should contain a citation note.
- Avoid too much abstraction, might be easier to simply have three, four
direct assessment implementations as opposed to supporting all possible
configurations. Some of those might never be used.
- Add receipt token output for MTurk or crowd sourcing integration.
- Do we need generic results output for easier extraction/processing?
- Export campaign status/stats via CSV export hook.
- Change URL scheme to be campaign-based, rather than type-based.